# Bhyve BVCP vs vm-bhyve



## MasterL (Oct 26, 2021)

Hi guys,
Soon I am going to setup a bhyve freebsd server. Reading about management alternatives most popular seems to be vm-bhyve
But BVCP sounds great.
It would be appreciated and helpful to me if anyone using bvcp could share about that web control panel for bhyve
Thank you


----------



## zirias@ (Oct 27, 2021)

Well, as there's no reply yet, I assume there aren't many people using it. Do you really need it? I could imagine such a "management GUI" is helpful with several servers all running lots of VMs. But if your setup is smaller, maybe keeping it simple would be the better approach indeed?



> *Optimized:* Every core features are optimized for one task: "manage virtual machines". Nothing more. Therefore the whole software less than 100MB.


Compare that to vm-bhyve's size of less than 250KB. Still, it supports everything you can do ATM with bhyve.

Just one hint, if you decide to install BVCP:


> *Storage Management:* Bhyve yet only supports .raw image disks, BVCP can create, attach, resize, detach, delete them as well.


I'd recommend to look into that. Bhyve supports virtual disks backed by ZFS zvols, which is a _very_ useful feature, so my question would be, is this really not usable with BVCP?


----------



## rootbert (Oct 27, 2021)

well bvcp seems like it is a binary blob where sources are not open ... that is a big no-go for me personally.


----------



## MasterL (Oct 27, 2021)

Zirias said:


> Well, as there's no reply yet, I assume there aren't many people using it. Do you really need it? I could imagine such a "management GUI" is helpful with several servers all running lots of VMs. But if your setup is smaller, maybe keeping it simple would be the better approach indeed?
> 
> 
> Compare that to vm-bhyve's size of less than 250KB. Still, it supports everything you can do ATM with bhyve.
> ...


Actually the server is for my nephew. His background with computers until now has been gaming. So I thought about giving him something like proxmox to start. Right now he understands web interfaces and has no command line experiences. The idea is to run bsd and linux servers and see what he likes and suits him. Also a docker vm since he is looking forward to setup his own minecraft server and a decent web page for his father’s business 
Maybe I could also start by teach him freebsd from zero


----------



## VladiBG (Oct 27, 2021)

Check CBSD framework or ClonOS


----------



## SirDice (Oct 27, 2021)

MasterL said:


> Right now he understands web interfaces and has no command line experiences.


Sounds like a good opportunity to kill two birds with one stone. Learn to use the command line _and_ learn to set up VMs. I've been using sysutils/vm-bhyve for a while now and love it. It's easy to use and understand, yet quite powerful in what it can do. 


```
root@hosaka:~ # vm list
NAME            DATASTORE  LOADER     CPU  MEMORY  VNC           AUTOSTART  STATE
case            default    bhyveload  4    4096M   -             Yes [4]    Running (2611)
jenkins         default    bhyveload  4    4096M   -             Yes [6]    Running (3127)
kdc             default    none       2    2048M   0.0.0.0:5901  Yes [2]    Running (2042)
lady3jane       default    bhyveload  4    8192M   -             Yes [3]    Running (2357)
debian          stor10k    uefi       2    4096M   -             No         Stopped
errol           stor10k    bhyveload  2    4096M   -             Yes [9]    Running (3921)
fbsd-test       stor10k    bhyveload  2    4096M   -             Yes [14]   Running (5233)
gitlab          stor10k    bhyveload  4    6144M   -             Yes [12]   Running (2233)
gitlab-runner   stor10k    bhyveload  4    4096M   -             Yes [13]   Running (2569)
kibana          stor10k    bhyveload  4    6144M   -             Yes [1]    Running (2102)
plex            stor10k    bhyveload  4    4096M   -             Yes [7]    Running (28761)
riviera         stor10k    bhyveload  2    4096M   -             Yes [11]   Running (4690)
sdgame01        stor10k    bhyveload  4    4096M   -             No         Stopped
tessierashpool  stor10k    bhyveload  2    4096M   -             Yes [5]    Running (3105)
ubuntu          stor10k    uefi       2    4096M   -             No         Stopped
wintermute      stor10k    bhyveload  4    4096M   -             Yes [10]   Running (4665)
```


----------



## zirias@ (Oct 27, 2021)

BTW:


> *UEFI Support:* Basically only supports UEFI installation, with its advantages and limitations. This is important due portability.


If I understand that correctly, this would be another "no-no" for me. Why should I bring up a VM with some emulated UEFI firmware if bhyveload or grub-bhyve would do fine? Less complexity, less risk some update (temporarily) breaks it.

IMHO, a GUI can probably help a lot for managing a large amount of VMs efficiently. It won't help at all if you can't solve problems at the commandline…


----------



## MasterL (Oct 27, 2021)

Zirias said:


> BTW:
> 
> If I understand that correctly, this would be another "no-no" for me. Why should I bring up a VM with some emulated UEFI firmware if bhyveload or grub-bhyve would do fine? Less complexity, less risk some update (temporarily) breaks it.
> 
> IMHO, a GUI can probably help a lot for managing a large amount of VMs efficiently. It won't help at all if you can't solve problems at the commandline…


Idk but the uefi thing might be because there is no-vnc support in the interface?


----------



## MasterL (Oct 27, 2021)

SirDice said:


> Sounds like a good opportunity to kill two birds with one stone. Learn to use the command line _and_ learn to set up VMs. I've been using sysutils/vm-bhyve for a while now and love it. It's easy to use and understand, yet quite powerful in what it can do.
> 
> 
> ```
> ...


Holy shit we just have 8G 
Way to go!


----------



## SirDice (Oct 27, 2021)

MasterL said:


> Holy shit we just have 8G


It's an old server I got after it was thrown away. It had 48GB and I bought an extra 48GB for dirt cheap on eBay (I think I spent $100 on it). Not a lot of power by today's standards but more than enough for my home lab. I can spin up quite a number of different machines to experiment with.


----------



## zirias@ (Oct 27, 2021)

MasterL said:


> Idk but the uefi thing might be because there is no-vnc support in the interface?


That might be a reason, but then, I'd argue it would be better to offer a serial console as an alternative instead…


----------



## DaVieS (Jan 23, 2022)

As an author I found this topic and I have some opinion about FreeBSD, Bhyve and its tools and BVCP.

.raw disk images are gives more flexibility and UEFI provides the installation future-proof so .raw images could be ported towards vmWare, VirtualBox, and so on, BVCP has API so you can fix things at commandline too. These are important things especially if you run a lot of production VMs that must be run.

Comparing BVCP to command-line scripts isnt fair, due they are using shared resources and depedencies, BVCP ships everything in a single package, which elliminiates disasters when you mess up the pkg and the ports for example when you partially upgrade your system then fails start anything due incompatibility. (happens a lot for me on dev machines) 
Scripts does not have built-in webserver and VNC tunneling, websockets and so on.

BVCP could be used as independently from the OS and packages, so* literally every BSD BOX could be turn into VM hypervisor*, you can completely reinstall your system without any trouble with VMs running on the background. At 2022 everyone suggested to be more energy-efficient (If not yet) due energy prices, this modell allowed me to build hassle-free "cluster" with machines that not primarily intended for virtualisation. 

I got a ton of response and help requests even if its childplay, people interested about virtualisation on Bhyve and FreeBSD, but do not have enough patience and knowledge or time to self manage their VMs. The short term consequence is a lot of people tried FreeBSD for first time because the graphical interface that BVCP offers, thats a win for me and for FreeBSD I guess.

Im aging as everyone and handling a ton of servers, now I have better things to do so graphical interfaces really made some part of my job easier, also I eliminated that annoys me the most, BVCP is running *unprivileged *unlike many others also it has enhanced security built-in and I get an email on log-in and even if the session stolen by an attacker. 

Despite all of this I have to say that Im using it for enterprise-level and production where many more rules forcefully applied, such as manageability. 
Like Investors, Leadershipment, who dont understand and care about serial console or even a pretty graphical terminal, what they would like to see is a graphical interface, including the fact if every IT personnel kidnapped they would like to manage theirself. O_O

By the way I tested every option for Bhyve including command line and graphical interfaces and I found out that FreeBSD still get a kernel panic on some scenarios like if you removing bridges on-the-fly, so it was a question how to use this in production? - i know, dont mess up, but .. - well I started writing scripts to avoid any potentional CHAOS and I ended up to the BVCP.

Worth to mention some existing tools especially which have remote access have some serious security concerns when I tested. Also I dont think so that the webinterface should be accessed by 'root' or running with root privilege.

So in conclusion I wanted a ProxMox for FreeBSD but with elevated security and I got it and shared it, however no free-lunch yet so its closed source, but it has real support in exchange for free of charge which is unlikely nowadays.

I understand the mean of the unixish feeling and really BSD part of my life since 2004 but the most of people scared out from FreeBSD due lack of human readable interface.


----------



## Lamia (Jan 24, 2022)

It is all here on GitHub - https://github.com/DaVieS007/bhyve-webadmin. Nothing wrong in setting it up for GUI access. I tried ClonOS back then. Then Bastille!!


----------



## rootbert (Jan 24, 2022)

Lamia said:


> It is all here on GitHub - https://github.com/DaVieS007/bhyve-webadmin. Nothing wrong in setting it up for GUI access. I tried ClonOS back then. Then Bastille!!



well i would not call it "It is all here on GitHub" since it is not open source but a binary file in a github repository ... just to mention it again so others are warned.


----------



## Lamia (Jan 24, 2022)

rootbert said:


> well i would not call it "It is all here on GitHub" since it is not open source but a binary file in a github repository ... just to mention it again so others are warned.


Thanks, I was speed reading and surfing. I thought the codebase was all there, no binary.


----------



## sko (Jan 24, 2022)

A huge gui that only a single hypervisor seems pretty useless/overkill... And the fact that it is a complete blackbox because there is no source available should be a big warning sign. If in 2 years the project gets abandoned and stops working after an OS upgrade you might end up with completely unusable VM images because they might do some weird/undocumented stuff with the image/config/loader/...

If you want/need FreeBSD as the base OS, go with tried-and-tested solutions like vm-bhyve.
If you are open to other hypervisors, you might want to take a look at smartOS (https://smartos.org) and especially (for simple GUI) Danube Cloud (https://danubecloud.org) as they support all kinds of virtualization from the GUI (bhyve, kvm, zones, lx-zones), not merely one.

BUT


MasterL said:


> Holy shit we just have 8G


In that case I wouldn't care much about (hardware) virtualization anyways, especially if you want to run some bloated OS like windows inside the VM(s) that absolutely needs 8GB+ for itself to run with bearable performance.
Depending on how many VMs you'd like to run and what else should run in jails (or zones), you should start at a bare minimum of 16GB or even 32GB RAM to also leave some headroom for ZFS to work with. A virtualization host under constant memory pressure is completely useless...


----------



## zirias@ (Jan 24, 2022)

sko said:


> If you want/need FreeBSD as the base OS, go with tried-and-tested solutions like vm-bhyve.


+1. It's simple, small, and works perfectly, at least for me.



sko said:


> Depending on how many VMs you'd like to run and what else should run in jails (or zones), you should start at a bare minimum of 16GB or even 32GB RAM to also leave some headroom for ZFS to work with.


Another +1, and just adding those are still "conservative" numbers.

My server here has 64GB, It has a 12TB ZFS pool (on 4 x 4TB disks), a small (1GB) FreeBSD VM for routing/firewall, another small (1GB) Linux VM for the WiFi controller and an 8GB Windows Server VM mainly for work and testing purposes.
Add some jails offering services (Active Directory DC with samba, SMB file server, Mail, Web, Asterisk PBX, ...) and I already had to say hello to the OOM killer while building packages with poudriere, using 8 parallel jobs


----------



## Lamia (Jan 24, 2022)

Seriously, vm-bhyve has no rival. I thought the said GUI was for the vm-bhyve. The TUI works sufficiently enough though. 
I shopped around back then when bhyve was gaining traction. Io-bhyve, cbsd, ClonOs, Bastille, etc. Today, it does the Linux jobs that jails cant among others (Windows etc). No more kfreebsd, De(vuan/bian)-in-jail talks. 

32GB RAM is a good start. It didn't take long before 16GB was exhausted.


----------



## sko (Jan 24, 2022)

Lamia said:


> Seriously, vm-bhyve has no rival. I thought the said GUI was for the vm-bhyve. The TUI works sufficiently enough though.


well, there also is iohyve which MAY be nice if you are already used to iocell/iocage for jails management. Anyhow - the subtle differences in syntax are enough to drive you crazy at times, so you can also just go for the broadly used and well documented vm-bhyve. The syntax isn't that huge and/or complex anyways...
both (vm-bhyve and iohyve) are purely shell-based and easily accessible for understanding/troubleshooting/extending functions, so just go with the one that suits your needs and "feels right for you".


----------



## Lamia (Jan 24, 2022)

Yes, each has its strengths just like ezjail, qjail then jail. Managing both - jail & VM/bhyve - with one well-accepted tool like or among those mentioned above is closer than we think.


----------

