# bhyve guest (win10) network access depends on startup procedure



## wearesix (Sep 25, 2017)

Hi there,

I discovered a strange behavior of the client network interface tap and an established bridge to the real networking device (em0). Each configuration (/etc/rc.conf...) is done as described in several howtos.
Client is a win 10 with Red Hat Virtio Eth Adapter, Driver 100.74.104.14100, date 19 jul 2017.
Older client-drivers are recommend (e.g. ...1.118), but has this has no effect.

Host:`uname -a`

```
FreeBSD  11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```

Creating the bridge _after_ starting bhyve will succeed, otherwise it fails.

It fails:
1.) creating the bridge with `ifconfig bridge0 create` ->bridge0 exists
2.) starting bhyve ... -s 5,virtio-net,tap0...
3.) Result: guest can't access network (DHCP is active, IP is class-b! with 255.255.0.0 as netmask!

It works:
1.) starting bhyve ... -s 5,virtio-net,tap0... 
2.) now creating the bridge with `ifconfig bridge0 create` -> bridge0 exists
3.) voilà, everything is fine. 

Any hints?
Thank you!


----------



## Eric A. Borisch (Sep 27, 2017)

I assume you're adding em0 and tap0 to the bridge, and just haven't listed that here.

Do you have the sysctl

net.link.tap.up_on_open=1

set? What about firewalls?


----------



## wearesix (Sep 27, 2017)

Hello,
i took this blog for my configuration:
https://www.ateamsystems.com/tech-blog/howto-windows-10-bhyve-w-freebsd-11/

firewall: nope; connection depends on the sequence, (please see above) - thats why i'm confused

sysctl.conf contains:

```
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.randompid=7152
hw.syscons.bell=0

kern.vt.enable_bell=0
#bhyve networking
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1
```

rc.conf:

```
#hostname="thinkpad"
keymap="de"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable

dumpdev="AUTO"
zfs_enable="YES"

hald_enable="YES"
dbus_enable="YES"
#wlans_iwn0="wlan0"
ifconfig_wlan0="WPA SYNCDHCP"
#Simple Login MGR
#slim_enable=yes
#networking bhyve
cloned_interface="bridge0 tap0"
ifconfig_bridge0="addm em0 addm tap0"
```

`sysctl -a | grep "tap"` :

```
net.link.tap.debug: 0
net.link.tap.devfs_cloning: 1
net.link.tap.up_on_open: 1
net.link.tap.user_open: 0
debug.if_tap_debug: 0
hw.psm.tap_timeout: 125000
hw.psm.tap_threshold: 25
hw.psm.tap_enabled: -1
```

Installed, but unused:
`vm version`

```
vm-bhyve: Bhyve virtual machine management v1.1-p8 (build 101073)
```

It is a fresh install of Freebsd, and i installed only things which seems to be necesary...
But something is missing :-(
Do i have to perform the basic setup for vm?
it's not configured until now.
my next trial: choosing another client-adapter (ne1000); report follows.


----------



## aragats (Sep 27, 2017)

It would be good to see your entire command line for starting `bhyve`.
Do you have `-s 0,hostbridge` option?


----------



## Eric A. Borisch (Sep 27, 2017)

As well as all of `ifconfig` output (mask actual IPs and MACs if you like) in “good” and “bad” states.


----------



## Eric A. Borisch (Sep 28, 2017)

And try this out:


ifconfig_em0="up promisc"
ifconfig_bridge0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable

dumpdev="AUTO"
zfs_enable="YES"

hald_enable="YES"
dbus_enable="YES"
#wlans_iwn0="wlan0"
#ifconfig_wlan0="WPA SYNCDHCP"
#Simple Login MGR
#slim_enable=yes
#networking bhyve
cloned_interface="bridge0 tap0"
ifconfig_bridge0="DHCP addm em0 addm tap0 up"


Not sure it will change anything, but worth a shot; I’ve had more consistency (I’m also doing pf filtering, which adds another variable) with the IP assigned to the bridge; I’m using a static IP, but hopefully this config will work, too. 

You also _do not_ need the forwarding sysctl set when using a bridge, unless you are forwarding between bridge0 and (commented out here) wlan0, as I recall.


----------



## wearesix (Sep 29, 2017)

Thank you for your assistance.
aragats:

```
bhyve -c 2 -m 4G -H -w -s 0,hostbridge -s 3,ahci-cd,null_dvd.iso -s 4,ahci-hd,w10_disk.img -s 5,virtio-net,tap0 -s 29,fbuf,tcp=0.0.0.0:5900,wait -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd win10
```

i figuered out the following:
When you startup the bridge before the tap-interface exits, tap0 won't become a member of the bridge automatically, look here:

`bhyve -c 2 -m 4G ....` 
-----------------
`ifconfig`

```
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:21:cc:6b:db:d6
        hwaddr 00:21:cc:6b:db:d6
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:80:37:ec:02:00
        hwaddr 02:80:37:ec:02:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ce:1b:cb:b9:00
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:0c:ae:f8:00
        hwaddr 00:bd:0c:ae:f8:00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 90733
```
$
`ps -xa`
...
34530  0  Is    0:00.01 sh
90733  0  I+    2:39.03 bhyve: win10 (bhyve)
31669  1  Ss    0:00.01 sh
...
-----------------
 tap0 isn't listed, result: no network-access

`ifconfig bridge0 addm tap0`
makes tap0 to a member of the bridge; also the client interface status changes instantanly from no network to fully functional network access!

`ifconfig`

```
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
        ether 00:21:cc:6b:db:d6
        hwaddr 00:21:cc:6b:db:d6
        inet6 fe80::221:ccff:fe6b:dbd6%em0 prefixlen 64 scopeid 0x1
        inet 192.168.178.61 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:80:37:ec:02:00
        hwaddr 02:80:37:ec:02:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ce:1b:cb:b9:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:a1:6b:f8:00
        hwaddr 00:bd:a1:6b:f8:00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 96071
```

is this behavior of the bridge as expected?


----------



## Eric A. Borisch (Sep 29, 2017)

If you’ve destroyed tap0 after its initial (on boot / init processing) creation, it won’t automatically become a member of bridge0 when it is recreated, if that’s what you are asking. (It should be a member after boot with the above rc.conf settings.) Opening and closing (but not destroying) the tap should leave it as a member.


----------

