# bhyve bridge networking does not work



## fnoyanisi (Feb 9, 2020)

Hi there,

I have a FreeBSD-RELEASE host machine (hostname = beastie) and I would like to run a FreeBSD-CURRENT on a bhyve VM (hostname = patch). The host was configured as described in the relevant section of the handbook (Section 21.7)

I am using iohyve as a bhyve wraper and I think association of the tap0 interface with VM is okay

```
root@beastie:~/vm # iohyve taplist
Listing all network taps:
zroot/iohyve/freebsd-current......tap0
```

My `re0` interface is assigned a static IP from the 192.168.1.0/24 pool and below is my `bridge0` and `tap0` configurations

```
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:86:c2:97:9d:00
        inet 192.168.1.64 netmask 0xffffff00 broadcast 192.168.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:5a:90:26:00
        groups: tap
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
The IP address of `bridge0` is assigned via DHCP (`dhclient bridge0`).

On the VM side, I have `vtnet0` interface along with `lo0`. I added 8.8.8.8 as the default nameserver.

This is where I am stuck...I want to configure internet access for the VM.

1 ) I followed the steps mentioned in the Bridged Network section of this tutorial but it does not work. I cannot set my default gateway hence getting no DHCPOFFERS (I tried setting the defaultgateway as 192.168.1.254, which is the gateway for the host, and IP address of `bridge0`, both failed). I used

```
route add default 192.168.1.254
```
2 ) Seems like I need some PF rules but cannot figure it out, any help with this would be appreciated.

Thanks


----------



## Emrion (Feb 9, 2020)

No pf is needed in configuration like this. Please post your /etc/rc.conf and /etc/sysctl.conf.

Also a `iohyve getall` of your VM and the content of /etc/rc.conf inside this VM.


----------



## Phishfry (Feb 9, 2020)

Did you add the sysctl entry like the handboot shows?
net.link.tap.up_on_open=1








						Chapter 23. Virtualization
					

Virtualization software allows multiple operating systems to run simultaneously on the same computer




					www.freebsd.org
				



The reason I ask is because I see 'no carrier' on your tap.


----------



## Phishfry (Feb 9, 2020)

I also feel that using an external program for bhyve is counter-productive.
For only a single VM you should really use the manual method so you get to know how things work.
Tools like iohyve and vm-bhyve are fine when you have many VM's but for a small number of VM's I feel its best to get a solid foundation. The handbook is very relevant and with the UEFI VM's it really is simple. Download the UEFI bhyve firmware and start the VM. I use /etc/rc.local to start my 5 VM's on bootup.
I also think bridge and tap are not the best route. Using a separate network interface for each VM is ideal. Pass them through to VM.
Currently I use Intel 4 port gigabit ethernet cards for my VM's.
Do you have any old ethernet cards laying around? That will simplify things in my opinon.


----------



## Emrion (Feb 9, 2020)

Phishfry said:


> I also feel that using an external program for bhyve is counter-productive.
> For only a single VM you should really use the manual method so you get to know how things work.
> Tools like iohyve and vm-bhyve are fine when you have many VM's but for a small number of VM's I feel its best to get a solid foundation. The handbook is very relevant and with the UEFI VM's it really is simple. Download the UEFI bhyve firmware and start the VM. I use /etc/rc.local to start my 5 VM's on bootup.
> I also think bridge and tap are not the best route. Using a separate network interface for each VM is ideal. Pass them through to VM.
> ...



sysutils/iohyve works very well for me. I'm very pleased with that one. So that isn't the problem.


----------



## fnoyanisi (Feb 9, 2020)

Hi. Thanks for the replies Emrion and Phishfry 

Below is my bhyve VM configuration along with the contents of /etc/sysctl.conf and /etc/rc.conf files (this is the host)

```
root@beastie:~ # iohyve getall freebsd-current
Getting freebsd-current iohyve properties...
bargs            -A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             0
con              nmdm0
cpu              1
description      FreeBSD-CURRENT
install          no
loader           bhyveload
name             freebsd-current
os               FreeBSD
persist          1
ram              1024M
size             35GB
tap              tap0
template         NO
utc              YES
vnc              NO
vnc_h            600
vnc_ip           127.0.0.1
vnc_tablet       NO
vnc_w            800
vnc_wait         NO
root@beastie:~ # sysctl net.link.tap.up_on_open
net.link.tap.up_on_open: 1
root@beastie:~ # cat /etc/sysctl.conf
# $FreeBSD: releng/12.1/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12

net.link.tap.up_on_open=1
root@beastie:~ # cat /etc/rc.conf
clear_tmp_enable="YES"
sendmail_enable="NONE"
hostname="beastie"
sshd_enable="YES"
powerd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

# static ip
# tap0 ip is for bhyve
defaultrouter="192.168.1.254"
ifconfig_re0="inet 192.168.1.59 netmask 255.255.255.0"

# Server load monitoring
netdata_enable="YES"

# web server
nginx_enable="YES"

# jails and byhve
# bridge0 and tap0 are required for bhyve
iocage_enable="YES"
cloned_interfaces="tap0 lo1"

pf_enable="YES"
pflog_enable="YES"
gateway_enable="YES"

# bhyve / iohyve
iohyve_enable="YES"
iohyve_flags="kmod=1 net=em0"
kld_list="nmdm vmm"
```

Nothing in fancy in VM's /etc/rc.conf file

```
root@patch:~ # cat /etc/rc.conf
clear_tmp_enable="YES"
sendmail_enable="NONE"
hostname="patch"
ifconfig_vtnet0="DHCP"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

root@patch:~ #
```

Even without any PF rules, I am not able to acquire any DHCP leases (I have an DHCP assigned IP for `bridge0` and I try to acquire a new IP from within the VM)


----------



## fnoyanisi (Feb 10, 2020)

Holy moly! Rebooting the guest without changing any settings solved the problem.... 

I spent all night trying to figure out this.

For future reference, the "Bridged Networking" option mentioned in this article works fine









						Bhyve Networking Options
					

Once in a while people on the #bhyve  IRC channel on freenode ask questions about bhyve networking configuration, i.e. how to configure thin...




					empt1e.blogspot.com


----------



## free-and-bsd (Dec 28, 2020)

Phishfry said:


> I also feel that using an external program for bhyve is counter-productive.
> ...


Not only that, but I also noticed that vm-bhyve, for example, makes things easy only with _generic_ configuration -- when all you need is just get bhyve up and running. 
The project's very goal seems to be exactly that: get that extra "load" from the shoulders of some user who is not going to use advanced options "in the very near future". Which seems to apply to a good deal of bhyve users -- ppl who resort to FreeBSD _because of _bhyve. 
Which shows, of course, the superiority of bhyve in some ways over other VM engines, that's true.

BTW, I DO have 2 nic in my machine... (Realtek, yes, but better than nothing). However, at my workplace I only have one cable, alas  . So I'll have to use the virtual tap network. But for me it's just about having a working MS Word for format compatibility. (Shame, too, because LibreOffice is superior in SO many ways actually. But "the world" doesn't want to consider that... sliding off topic here, sorry).


----------

