# Upgrading 13.0-RELEASE-p11 to 13.1-RELEASE breaks bhyve tap interface inside jails



## Yze (May 26, 2022)

Hi everyone,

another surprise occurred after upgrading 13.0R to 13.1-RELEASE. The linux VM stopped starting that is using 3 tap devices to talk to the host. My best guess is that access to tap interfaces changed but didn't yet find a way to give permission to the jail.

Jail used to start with:


```
grub-bhyve -m device.map -r host -d /vmm/lnx1v -c /dev/nmdm3A -M 6GB lnx1v-jail
bhyve -c 2 -A -H -P -m 6GB -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/md42 -s 6,virtio-blk,/dev/zvol/tank/vm/lnxdisk0 -s 3,virtio-net,tap0 -s 4,virtio-net,tap1 -s 5,virtio-net,tap2 -lcom1,/dev/nmdm3A lnx1v-jail

Could open socket
device emulation initialization error: Protocol not supported
```


```
jail.conf snipped:

bee {
    devfs_ruleset = 27;
    allow.vmm;
    allow.raw_sockets;
    persist;
    allow.socket_af;
    host.hostname = "bee";
    path = /tank/bee;
}
```

tried to add allow.raw_sockets as well allow.socket_af as it appears to be socket protocol related, but didn't allow bhyve to work with it.

sysctl that previously worked:


```
net.link.tap.up_on_open=1
net.link.tap.user_open=1
```

and devfs.rules for bhyve;:


```
[devfs_rules_bhyve_jail=27]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path vmm unhide
add path vmm/* unhide
add path vmm.io unhide
add path vmm.io/* unhide
add path tap* unhide
add path zvol unhide
add path zvol/tank unhide
add path zvol/tank/vm unhide
add path zvol/tank/vm/* unhide
add path nmdm* unhide
add path 'md*' unhide
add path 'md*' mode 0777
#add path mem unhide
#add path kmem unhide
#add path pci unhide
#add path io unhide
```

Wondering, if anyone in the field also runs bhyve inside jails using tap and got success in 13.1-RELEASE to start them (again). Any hints appreciated.


----------



## Erichans (May 26, 2022)

What exactly did you upgrade from 13.0-RELEASE to 13.1-RELEASE ?


----------



## Yze (May 26, 2022)

Erichans said:


> What exactly did you upgrade from 13.0-RELEASE to 13.1-RELEASE ?


sorry, amd64 arch if thats relevant - FreeBSD 13.1-RELEASE fc952ac22 EU amd64


----------



## sku1d (May 29, 2022)

I have hot-fixed an issue, which could be related.

After the update I was unable to start my openvpn in a jail with an error message `Cannot open TUN/TAP dev /dev/tun100: No such file or directory (errno=2)`. It turned out, that this was due to missing device notes under `/dev/`- no `/dev/tun100` there, even though I have configured it in the hosts `devfs.conf`. I tried to further change my `/etc/devfs.conf`, but none of these adaptions showed the expected effect on jails configured with a `devfs_ruleset` in `/etc/jail.conf`, which used to work before the upgrade.

My current workaround is, that I copied my contents from `/etc/devfs.conf` to `/etc/defaults/devfs.conf` and after a restart of the machine it was working again. This is a hotfix rather then an ideal solution. I hope it helps someone to save that amount of time which is required to find out what has changed there


----------



## Yze (Jun 5, 2022)

hmm the devfs seems to work so far with my rules - e.g. /dev/tap0 shows up. Running truss shows it "dies" when it tries to use fnctl syscall against that tap device and gets "unsupported protocol" back from kernel. Not much we can do I would assume. To me that's a kernel bug - I could now reproduce that in another lab box and have to see how I can report that. Running bhyve in jail seems is nobody doing or they have all systems which do not use networking...


----------



## zirias@ (Jun 5, 2022)

Yze said:


> Running bhyve in jail seems is nobody doing


Just out of curiosity: What's the point? I mean, most of the hypervisor code is in-kernel anyways?

Doesn't mean it shouldn't work, of course. I just try to understand what could be the benefit...


----------



## Yze (Jun 6, 2022)

I do agree, as it does not separate the actual VM execution into the jail instance. We used bhyve like this from day 1, so far we never had issues with that design.
However, we run all hosts itself with bare minimum on read-only storage, everything is spawned in jails using another storage. Base is ideal without any pkg installed at all - pure "installworld". For the lab - the workaround atm is to use chroot instead of jail (which obviously works) but it would not hurt to keep the design with jail.


----------

