# Problem with multiple bridges on interface with VLAN



## Drizzt321 (Feb 26, 2021)

So I have a few VMs via churchers vm-bhyve, and trying to put a new VM on with a VLAN. The Unifi switch port is configured as a trunk port. So starting out, here's the interfaces I have.


```
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether d0:50:99:dd:49:c9
        inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 92:35:c4:3c:e8:d0
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-pihole-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:4a:1a
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 83050
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-unifi-controller-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:ba
        inet6 fe80::5a9c:fcff:fe10:ffba%tap2 prefixlen 64 tentative scopeid 0xa
        groups: tap vm-port
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

Everything works just fine. So I then create the bridge (virtual-switch in vm-bhyve parlance), for the VLAN, and boot up the VM. I end up with these interfaces.


```
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-iot-igb0.30
        options=200001<RXCSUM,RXCSUM_IPV6>
        ether d0:50:99:dd:49:c9
        inet6 fe80::d250:99ff:fedd:49c9%igb0.30 prefixlen 64 scopeid 0x7
        groups: vlan vm-vlan viid-45833@
        vlan: 30 vlanpcp: 0 parent interface: igb0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-iot: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:2d:aa:58:d8:1b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: igb0.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 55
        groups: bridge vm-switch viid-97c20@
        nd6 options=1<PERFORMNUD>
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-home-assistant-0-iot
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:b1
        inet6 fe80::5a9c:fcff:fe10:ffb1%tap0 prefixlen 64 tentative scopeid 0x8
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 4573
```

After the VM finishes starting up, nothing, the VM can't get a DHCP from the VLAN or ping anything. Doing tcpdump on both the router I see the DHCP request and response on the correct VLAN (30), and on the host igb.30, I see the DHCP request, but no response.

From https://github.com/churchers/vm-bhyve/issues/390#issuecomment-785537662 someone else mentions when the bridge on the untagged interface, suddenly the bridge on the VLAN interface starts passing traffic fine. I try the same, shut down the VMs on the public (untagged) bridge, still nothing, and then I destroy the bridge. Suddenly the DHCP works fine, as well as pinging. I then create the bridge on the untagged interface again, and neither ping nor DHCP works correctly anymore.

I then destroy the untagged bridge, and it's back to working again. 
I then try and create another VLAN bridge (id 40), and while the bridge on VLAN 30 VM still works, another VM up on the VLAN 40 bridge, and both VLAN bridges (30 and 40) work just fine, no problems with DHCP or ping.
If I create the untagged bridge again, both VMs on VLAN 30 and 40 stop working. 

It's as if the untagged bridge swallows all traffic intended for the VLAN bridges or interfaces. 

I'm running v12.2-RELEASE*.*


----------

