# Howto: Bhyve, using OpenBSD as main firewall in FreeBSD



## t1066 (Feb 20, 2015)

My idea is to run the latest OpenBSD in FreeBSD as a guest using bhyve. Then pass a network interface to the OpenBSD guest using PCI pass through. So we can use the OpenBSD guest as the firewall of the host FreeBSD system. I got this idea from reading the following thread. I had learned a lot from the handbook, wiki and the article Intel Wifi Via Bhyve on FreeBSD. And I had also learned from many other sources. Let us now proceed to the howto.

First, we identify the network interface that we would like to pass through to OpenBSD. In the following example, I use an Intel 82576L network interface. Identifying the interface:

```
# pciconf -lv

snippet
em1@pci0:5:0:0: class=0x020000 card=0x00000000 chip=0x10d38086 rev=0x00 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82574L Gigabit Network Connection'
  class  = network
  subclass  = ethernet
snippet
```
Here I will use em1@pci0:*5:0:0*.

Now add the following lines to /boot/loader.conf.

```
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
pptdevs="5/0/0"
```
We need to reboot for PCI pass through to work. During boot, instead of attaching the em driver to the interface, we see the following.

```
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
ppt0 port 0xb000-0xb01f mem 0xf7a00000-0xf7a1ffff,0xf7a20000-0xf7a23fff irq 19 at device 0.0 on pci5
```
Next, we need a working image of OpenBSD. Because of current limitation of bhyve, we have to obtain it using other virtualization software. Moreover, as we are interested in using OpenBSD as a firewall, we need only to install the basic components of OpenBSD. See post #4 for how to install OpenBSD in bhyve.
Also, remember to check /etc/ttys in OpenBSD to allow serial console. Otherwise, booting the OpenBSD image will stuck at the TIME and DATE line.

After obtaining such an image, say, obsd56.img, I put it in /root/bhyve/openbsd. Also create the following two files:

```
1. /root/bhyve/openbsd/obsd56.map:

(hd0) /root/bhyve/openbsd/obsd56.img

2. /root/bhyve/openbsd/obsd56.in:

kopenbsd -h com0 -r sd0a (hd0,openbsd1)/bsd
boot
```
Now, install sysutils/grub2-bhyve and prepare the network interfaces.

```
# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
# ifconfig bridge0 create
# ifconfig bridge0 addm tap0 up
# ifconfig bridge0 inet 192.168.0.1 netmask 255.255.255.0
```
Run the following commands:

```
# grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r hd0 \
> -M 256M obsd56 < /root/bhyve/openbsd/obsd56.in > /dev/null
# bhyve -m 256M -A -H -P -s 0:0,amd_hostbridge -s 1:0,lpc \
> -s 2:0,passthru,5/0/0 -s 3:0,virtio-net,tap0 \
> -s 4:0,virtio-blk,/root/bhyve/openbsd/obsd56.img \
>  -l com1,stdio -W obsd56
```
You should be able to see OpenBSD booting. Log into the OpenBSD system at the login prompt. In OpenBSD, you should see two network drivers, vio0 and em0. First, create the following file /etc/hostname.vio0

```
inet 192.168.1.2 255.255.255.0
```
Now run `sh /etc/netstart` in OpenBSD.

Test the connection between the host and the guest. In FreeBSD, run `ping -c 2 192.168.1.2` and in OpenBSD run `ping -c 2 192.168.1.1`.

If there is no problem, set up em0 as your main gateway in OpenBSD. See the OpenBSD FAQ for further information. Then set up firewall in the guest OpenBSD. Finally, rerun `sh /etc/netstart`.

Return to the host FreeBSD system. We need to add the following lines to /etc/rc.conf.

```
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="inet 192.168.1.1 netmask 255.255.255.0 addm tap0 up"
defaultrouter="192.168.1.2"
```
and add net.link.tap.up_on_open=1 to /etc/sysctl.conf.
We also have to make changes to /etc/resolv.conf if you are using dhcp. We can test our setting here by restarting network in FreeBSD. A simple test is to look up www.freebsd.org. And then ping www.freebsd.org.

Finally, create the following file, /etc/rc.local.

```
#/bin/sh
/usr/local/sbin/grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r hd0 \
> -M 256M obsd56 < /root/bhyve/openbsd/obsd56.in > /dev/null
/usr/sbin/bhyve -m 256M -A -H -P -s 0:0,amd_hostbridge -s 1:0,lpc \
> -s 2:0,passthru,5/0/0 -s 3:0,virtio-net,tap0 \
> -s 4:0,virtio-blk,/root/bhyve/openbsd/obsd56.img \
>  -l com1,nmdm0A -W obsd56
```
Note that we replace stdio with nmdm0A in the above script. Reboot the host system to test that it works. Enjoy using a OpenBSD firewall in FreeBSD.


----------



## devil_devil (Feb 20, 2015)

Excellent howto. 

Congratulations!


----------



## BSDBernd (Feb 20, 2015)

Wow great, I hoped my thread would be of use and now it gets even cited, I am honored . Bhyve is a masterpiece. I cannot imagine what will
be possible in the future using that .


----------



## grehan@ (Feb 22, 2015)

To clarify:



t1066 said:


> Because of current limitation of bhyve, we have to obtain it using other virtualization software.



bhyve has been able to boot the OpenBSD install kernel (bsd.rd) for quite a while now. For example, to install from the 5.6 ISO image, you would put the following line into the obsd56.map file


```
(cd0) /path/to/install56.iso
```

The grub-bhyve command line would be


```
grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r cd0 -M 256M obsd56
kopenbsd -h com0 /5.6/amd64/bsd.rd
boot
```

bhyve is started with the same command line with one addition:


```
-s 5:0,ahci-cd,/path/to/install56.iso
```

Later,

Peter.


----------



## kevingnet (Apr 28, 2016)

Having some issues with bhyve, pci passthru mode isn't working, the -S switch is not available except in CURRENT but could not compile the grub2-bhyve port, due to other compiler issues in freebsd ports, but the compiler itself can build small binaries.


----------



## gofer_touch (Apr 28, 2016)

Hopefully this makes it to BSD Now, there is very wide use for this.


----------



## kevingnet (Apr 29, 2016)

I was able to use passthru before but with a freebsd guest.


----------



## zirias@ (Apr 29, 2016)

The idea isn't totally new ... I'm doing exactly the same thing on my home server for years, but using XEN and Linux. But great to hear it works with bhyve, as I'm planning to use FreeBSD on my new server, and this was what I had in mind anyways. I didn't think about using OpenBSD for the firewall VM, but this might be a good idea as well, so thanks, I'll come back soon and give it a try


----------



## ikevin8me (Aug 13, 2017)

I'd like to ask/clarify:

If the network card is passed through directly to OpenBSD, how do I assign IP addresses (say, 192.168...) to jails? (Normally, I'd use the interface, like em0, to assign to jails.)

Please clarify what is the right way to assign IP address to jails if the network card is no longer visible to the FreeBSD operating system.

Thanks!


----------



## t1066 (Aug 14, 2017)

ikevin8me said:


> I'd like to ask/clarify:
> 
> If the network card is passed through directly to OpenBSD, how do I assign IP addresses (say, 192.168...) to jails? (Normally, I'd use the interface, like em0, to assign to jails.)
> 
> ...


I do not use jail. But I would guess you use the interface tap0 for jails. Also, I had encountered a problem that hot swap does not work when vmm.ko is loaded.


----------

