# bhyve with vlans - host and vm can't pass traffic



## Scott O'Connell (Apr 21, 2015)

I'm very new to bhyve and am having an issue.  I'm trying to get VM's and VLAN's working.

I'm able to get VLAN's working in a VM, but the VM and the VMHOST, can't communicate with each other on the same vlan.

Using 10.1-RELEASE-p9 for both VMHOST01 and DEV.  Upstream from the VMHOST on lagg0 is a Cisco 3750G.

*VMHOST01 before starting VM:*

```
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
  ether f0:1f:af:dd:2e:c5
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
  ether f0:1f:af:dd:2e:c5
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
  ether f0:1f:af:dd:2e:c5
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: active
  laggproto lacp lagghash l2,l3,l4
  laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=103<RXCSUM,TXCSUM,TSO4>
  ether f0:1f:af:dd:2e:c5
  inet 10.0.1.17 netmask 0xffffff00 broadcast 10.0.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: active
  vlan: 100 parent interface: lagg0
```
*VMHOST after starting VM (added tap0 & bridge0):*

```
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:70:71:1d:00
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: no carrier
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  ether 02:d3:e4:02:03:00
  nd6 options=1<PERFORMNUD>
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 6 priority 128 path cost 2000000
  member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
  ifmaxaddr 0 port 4 priority 128 path cost 10000
```
Note that the _"status: no carrier"_ is because I hadn't brought up the VM yet.  It properly changes to the following after the VM is started:

```
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:70:71:1d:00
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: active
  Opened by PID 70827
```
*VM:*

```
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
  ether 00:a0:98:2b:34:37
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet 10Gbase-T <full-duplex>
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  ether 00:a0:98:2b:34:37
  inet 10.0.1.6 netmask 0xffffff00 broadcast 10.0.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet 10Gbase-T <full-duplex>
  status: active
  vlan: 100 parent interface: vtnet0
```
With this configuration, both VMHOST01 and DEV can communicate anywhere, EXCEPT to each other using their IP on VLAN100.

The ultimate goal is to have more than one VLAN presented to the VM, whether it exists on the VMHOST or not.  

Where did I go wrong?

Thanks in advance,
scotto


----------



## usdmatt (Apr 22, 2015)

Interesting problem. Might be something for the freebsd-net@ mailing list as it could do with people who understand how packets (and importantly vlan packets) are handled through the network stack.

Out of curiosity, does disabling hardware vlan tagging on the lagg interface make any difference?


----------



## Scott O'Connell (Apr 22, 2015)

Thanks for the reply and the ideas to move forward.

I tried to remove vlan tagging on the lagg interface as suggested, but it appears -vlanhwtag only works on physical interfaces.  

```
root@vmhost01[509]# ifconfig lagg0 -vlanhwtag
ifconfig: -vlanhwtag: Invalid argument
```

I'll look into the freebsd-net mailing list and see if I can make some progress.

Thanks again.


----------



## SemFLY (Oct 1, 2016)

Hi 

I see, thread start at 2015 year in april month, but may be this information help to somebody.
How work with VLAN's and bhyve VM

*On Host *FreeBSD 11-Release
_vlan420 - have 4(four) real ip address
Network 200.xxx.xxx.120/29_


```
# bhyve -c 2 -s 7,fbuf,tcp=0.0.0.0:5900 \
-s 0,hostbridge -s 3,ahci-hd,fbsd_hdd.img \
-s 4,ahci-cd,FreeBSD-11.0-RELEASE-amd64-disc1.iso \
-s 10,virtio-net,tap0 \
-s 31,lpc -l bootrom,BHYVE_UEFI.fd \
-m 2G -H -w fbsd
```


```
# ifconfig

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether f0:79:59:62:1a:19
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:97:2d:32:0d:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000

vlan420: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1<RXCSUM>
        ether f0:79:59:62:1a:19
        inet 200.xxx.xxx.125 netmask 0xfffffff8 broadcast 200.xxx.xxx.127
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 420 vlanpcp: 0 parent interface: re0
        groups: vlan
```


*guest* FreeBSD 11-Release

```
# ifconfig
vtnet0:
--- listing cut ---

vlan420: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1<RXCSUM>
        ether f0:79:59:62:1a:19
        inet 200.xxx.xxx.126 netmask 0xfffffff8 broadcast 200.xxx.xxx.127
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        vlan: 420 vlanpcp: 0 parent interface: vtnet0
        groups: vlan
```

Don't foget add default route on guest

```
# route add default 200.xxx.xxx.121
```

Your VM machine will be have real IP


----------



## SemFLY (Oct 3, 2016)

Hmm... have the same problem... i think work is good, but i try ping from host to vm and from vm to host... not working
If host and vm have the same VLAN(example - vlan100).

vlan100 on host and on guest
Network 192.168.11.0/24

host 192.168.11.212
guest 192.168.11.214

ping from host to vm
and
ping from vm to host

can't communicate with each other on the same vlan

What ideas ?


----------



## dehrmann (Jun 23, 2017)

Any updates or workaronuds? I think I ran into the same issue. I narrowed it down to the guest getting ARP requests, but the host never seeing the replies. This is with 11.0-RELEASE-p9.


----------



## tcn (Nov 3, 2017)

Has anyone been able to figure this one yet?  I am facing the very same issue where ARP replies are not reaching the VM.
FreeBSD host is able to get to the VM; VM sees host; FreeBSD can reach the router on the VLAN but the VM can't.


----------

