# bhyve



## Berserker79 (Mar 31, 2021)

First off, long time user recently switched back to FreeBSD from linux. I have been intrigued by a distro named Qubes. It uses Xen as a hypervisor to create seperate doms for things such as work, home, internet, etc. I am going to try to use bhyve in a like manner. I searched the forums and did not find any similar posts. Any help or observation from more knowledgeable users will be greatly appreciated. Please don't beat me up with the likes of FreeBSD is not linux, or FreeBSD is not a distro. I am totally aware of all of this, and their nuances. I'm just looking for friendly help, and good discussions.


----------



## zirias@ (Mar 31, 2021)

So? Then use bhyve, it works well for me with FreeBSD, Linux and Windows guests. But just a little hint: You don't always need a VM for separation. For most usecases, if the "guest" is FreeBSD as well, jails will do (and if you need network separation, have a look at VNET jails).


----------



## scottro (Mar 31, 2021)

I have a little page on using vm-bhyve.  which is a nice tool for managing bhyve virtual machines. Probably more importantly, it has links to the bhyve
quick start guide. You don't have to use vm-bhyve or any of the various management tools that are around, but I found it made my life easier.


			Windows on vm-bhyve
		


The page is mostly aimed at running Windows, but should hopefully give hopeful information for running Linux as well.  And the quickstart page that's linked gives an example of a FreeBSD guest.


----------



## diego (Apr 1, 2021)

Glad to hear more people are interested on Bhyve.......

First place ---> official documentation 21.7. FreeBSD as a Host with bhyve
Go this room -->  Emulation and virtualization. with posts covering bhyve
Another instesting post --> bhyve where to start
I would recommend to use vm-bhyve (much easier)
Welcome to the party


----------



## Berserker79 (Apr 1, 2021)

Thank you Zirias, scottro, and diego for your insight. I appreciate it, and will begin to digest the info you provided.


----------



## zirias@ (Apr 1, 2021)

I'll elaborate a bit on the suggestion to also consider VNET jails: With VNET, jails give you a "userspace virtual machine"; separation should be comparable. You can use epair(4) interfaces for the jails and plug them into a bridge(4) to have them in a separate virtual network, just like you would with the tap(4) interfaces used for a byhve vm.

The advantage of a jail is: It's running on the same kernel, so there's less overhead, it can just use a reserved part of the host's filesystem (and if necessary, you can delegate it ZFS datasets to manage them inside the jail) and shares the RAM, so no need to "wire" a substantial part of RAM (that can't be swapped) just for the virtual machine. You can run older versions of FreeBSD in a jail, as long as your kernel has compatibility enabled (which is the default), or even Linux, if the kernel has Linux compatibility (also the default), but I wouldn't recommend the latter, as the compatibility with Linux isn't perfect and there are pitfalls.

Of course, if you need a different OS, or only a newer FreeBSD version, or a kernel compiled with different options, you can't use a jail and need bhyve. But you can easily mix both, that's what I'm doing. Maybe to give some ideas, I'll share part of my setup:

I'm using a bhyve VM for routing and firewalling. If you can afford to run a separate phyiscal box for firewalling, consider doing so, because no VM or jail can ever 100% guarantee no "break-out" is possible, so if an attacker breaks out of a different jail or VM on the same host, they can also control the firewall. But at least, this risk is relatively small, and putting the firewall in a VM is the next best thing you can do (better than firewalling directly on the host).

So, this firewall VM needs bhyve, to be able to "PCI pass-through" all physical NICs to it. It also has several virtual network interfaces, and the host-side tap(4) interfaces are plugged into different bridges to form different virtual subnets.

Then, I also have e.g. a Windows machine in bhyve.

For most other things, I use VNET jails here. For example, one jail is an AD DC (with samba), another is a samba fileserver (for windows clients to access home etc.), yet another one provides a webserver, etc. – using bhyve for all these would be unnecessary overhead 

Finally, I support diego's suggestion to use sysutils/vm-bhyve, it's a nice little management tool. If you use ZFS, consider creating a sparse zvol to back your virtual harddisk – this is also supported by vm-bhyve, just look at the docs. If you make sure the guest has TRIM enabled, you will see the zvol shrink again when blocks on the virtual harddisk aren't used any more.

And yet another hint: You should use "virtio" drivers in your guests for disk and network, this should give the best performance. FreeBSD and Linux come with the necessary drivers, for Windows guests you can install them from here: https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md


----------

