# VLAN tagged interfaces in bridges



## bautsche (Feb 15, 2019)

Hi All.
I'm trying to add a vlan tagged interface to a bridge.
I've got an interface called re0 on the server, this is part of bridge0 and bridge0 has an IP address that pings just fine.

I now create a vlan tagged interface thus:
ifconfig re0.33 create vlan 33 vlandev re0 up

If I put an IP address on that, it pings just fine.

OK, without that IP address, I now create bridge1:

```
ifconfig bridge create
ifconfig bridge1 addm re0.33
```
If I now put an IP on that bridge instead of re0.33, it does not ping.

If I do a broadcast ping from another host on that network thus (Solaris system issuing the ping):
ping -sn 192.168.33.255

I can see packets arriving if I `tcpdump -i re0.33` and if I `tcpdump -i bridge1`
However, on neither interface do I see any pings coming in when I ping it's own address (in this case 192.168.33.20).
The Solaris system issuing the pings has learned the arp address of the bridge though:

```
root@gaspra # arp -an | grep 192.168.33.20
net1   192.168.33.20        255.255.255.255          02:a7:91:b6:3a:01
```
If I `tcpdump -i bridge1`, I do get some packets, but not any echo requests:

```
root@bianca # tcpdump -i bridge1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:05:26.081185 ARP, Request who-has 192.168.33.20 (Broadcast) tell juliet-punchin.swangage.co.uk, length 46
11:05:26.081197 ARP, Reply 192.168.33.20 is-at 02:a7:91:b6:3a:01 (oui Unknown), length 28
11:05:38.201079 IP6 fe80::7285:c2ff:fea6:583c > ff02::2: ICMP6, router solicitation, length 16
11:06:04.079441 ARP, Request who-has 192.168.33.20 (Broadcast) tell juliet-punchin.swangage.co.uk, length 46
11:06:04.079464 ARP, Reply 192.168.33.20 is-at 02:a7:91:b6:3a:01 (oui Unknown), length 28
11:06:17.588644 ARP, Request who-has 192.168.33.20 (Broadcast) tell gaspra-punchin.swangage.co.uk, length 46
11:06:17.588665 ARP, Reply 192.168.33.20 is-at 02:a7:91:b6:3a:01 (oui Unknown), length 28
```
So as you can see, the FreeBSD system does get ARP requests and does respond to them, but the pings never get there....


And here's the ifconfig output on the FreeBSD system being pinged:

```
root@bianca # ifconfig -a
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 80🇪🇪73:63:5c:48
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a7:91:b6:3a:00
        inet 192.168.140.85 netmask 0xffffff00 broadcast 192.168.140.255 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge 
        nd6 options=9<PERFORMNUD,IFDISABLED>
re0.33: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 80🇪🇪73:63:5c:48
        inet6 fe80::82ee:73ff:fe63:5c48%re0.33 prefixlen 64 scopeid 0x4 
        groups: vlan 
        vlan: 33 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a7:91:b6:3a:01
        inet 192.168.33.20 netmask 0xffffff00 broadcast 192.168.33.255 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0.33 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        groups: bridge 
        nd6 options=9<PERFORMNUD,IFDISABLED>
root@bianca #
```
There are no firewalls involved at all here.

Any pointers greatly appreciated.

Thanks.
Eric


----------



## SirDice (Feb 15, 2019)

How is the switch port configured? Make sure you're allowing VLAN tagged packets and have setup a correct native VLAN. Your re0 (via bridge0) is on the native VLAN (no VLAN tagging) and re0.33 is tagged. So you need to allow both types of traffic on the same port.


----------



## bautsche (Feb 15, 2019)

The switch port works properly. See also my comment above:


> I now create a vlan tagged interface thus:
> ifconfig re0.33 create vlan 33 vlandev re0 up
> 
> If I put an IP address on that, it pings just fine.



So that side of it all works. It just stops working if I put it in a bridge.

Eric


----------



## SirDice (Feb 15, 2019)

I have VLAN tagged traffic on my bridges (used for VMs) so I know it works. But none of my bridges have an IP address, thus my configuration is a little different from yours. 


```
root@hosaka:~ # ifconfig vm-servers
vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether f2:05:74:cd:05:3c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: lagg0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 55
        groups: bridge vm-switch viid-d5539@
        nd6 options=1<PERFORMNUD>
root@hosaka:~ # ifconfig lagg0.11
lagg0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-servers-lagg0.11
        ether 00:25:90:f1:58:39
        inet6 fe80::225:90ff:fef1:5839%lagg0.11 prefixlen 64 scopeid 0x8
        groups: vlan vm-vlan viid-8bf4d@
        vlan: 11 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@hosaka:~ # ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:f1:58:39
        laggproto lacp lagghash l2,l3,l4
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```


----------



## bautsche (Feb 15, 2019)

Hmm, that's interesting.
How do you configure this? via a script? in rc.conf?
And are you able to share that?
I wonder if the issue with my config is that both bridges are in the same "group", i.e. "bridge"? (but a test doesn't seem to bear that out).


----------



## bautsche (Feb 15, 2019)

The vlan tagging definitely works (it was the first thing I tested):


> I now create a vlan tagged interface thus:
> ifconfig re0.33 create vlan 33 vlandev re0 up
> 
> If I put an IP address on that, it pings just fine.


----------



## SirDice (Feb 15, 2019)

I'm using sysutils/vm-bhyve which creates them.

```
root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
```

But it's just a bunch of complex shell scripts that use the standard FreeBSD commands to set them up.

While playing around with STP and bridge(4) I did find some unexpected oddities. It turns out that a bridge(4) is hooked into the interface _before_ the interface itself gets a chance to process the packets. In your case I think bridge0 might be capturing the packets before they get passed to re0.33.


----------



## bautsche (Feb 15, 2019)

Hmm, I'm using XEN in the hopes of being able to run Solaris VMs.... I suspect the bhyve bits are going to clash....


----------



## bautsche (Feb 15, 2019)

OK, tried it anyway, and it turns out it does not conflict with the XEN bits.
I did this:


```
pkg install vm-bhyve

zfs create -o mountpoint=/export/vm/bhyve zroot/vm/bhyve
zfs set compression=on zroot/vm/bhyve
echo "vm_enable=\"YES\"" >> /etc/rc.conf
echo "vm_dir=\"zfs:zroot/vm/bhyve\"" >> /etc/rc.conf

vm init
```

And then created my switch on vlan33:
`vm switch create -i re0 -n 33 -a 192.168.33.20/24 vlan33`


Alas the result is the same as before: The IP address 192.168.33.20 does not ping... :-(

ifconfig now looks like this:

```
root@bianca # vm switch create -i re0 -n 33 -a 192.168.33.20/24 vlan33
root@bianca # vm switch list
NAME    TYPE      IFACE      ADDRESS           PRIVATE  MTU  VLAN  PORTS
vlan33  standard  vm-vlan33  192.168.33.20/24  no       -    33    re0
root@bianca # ifconfig -a
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 80:ee:73:63:5c:48
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a7:91:b6:3a:00
        inet 192.168.140.85 netmask 0xffffff00 broadcast 192.168.140.255 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge 
        nd6 options=9<PERFORMNUD,IFDISABLED>
vm-vlan33: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 7e:68:06:ab:55:e9
        inet 192.168.33.20 netmask 0xffffff00 broadcast 192.168.33.255 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0.33 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
        groups: bridge vm-switch viid-fd087@ 
        nd6 options=1<PERFORMNUD>
re0.33: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-vlan33-re0.33
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 80:ee:73:63:5c:48
        inet6 fe80::82ee:73ff:fe63:5c48%re0.33 prefixlen 64 scopeid 0x5 
        groups: vlan vm-vlan viid-0fac1@ 
        vlan: 33 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@bianca #
```


----------



## phoenix (Feb 15, 2019)

You really shouldn't mix tagged and untagged vlans on the same physical interface.  While it usually works on FreeBSD (and breaks things horribly on Linux), it doesn't always work the way you think it should.  

Tag all your vlans on the switch port (make it a trunk port, not a hybrid port).

Don't use re0 directly.  Create vlan tags for everything:  re0.1, re0.33

Use those in the bridges:  bridge0 --> re0.1; bridge1 --> re0.33

See if that works.


----------



## bautsche (Feb 15, 2019)

That's not really an option.... That would stop me from doing a network boot for the system. :-(

I could test if it makes a difference of cause, but it'll have to be over the weekend now...

P.S. it works without any issues on my Linux systems as well as on my Solaris systems. I do appreciate that a tcpdump of the underlying non-tagged device shows up tagged packets (as in with the tag), too, but I can live with that and it is considered in my security concept.
E.g.

```
root@miranda # snoop -d net0
Using device net0 (promiscuous mode)
oberon.swangage.co.uk -> miranda      TCP D=22 S=58886 Ack=530107858 Seq=896572573 Len=0 Win=64436 Options=<nop,nop,tstamp 418865960 489994884>
dns-ext.swangage.co.uk -> thalassa.swangage.co.uk HTTP C port=33050 
VLAN#40: dns-ext.swangage.co.uk -> thalassa.swangage.co.uk HTTP C port=33050 
VLAN#40: thalassa.swangage.co.uk -> dns-ext.swangage.co.uk HTTP R port=33050 
thalassa.swangage.co.uk -> dns-ext.swangage.co.uk HTTP R port=33050 
dns-ext.swangage.co.uk -> thalassa.swangage.co.uk HTTP C port=33050 
VLAN#40: dns-ext.swangage.co.uk -> thalassa.swangage.co.uk HTTP C port=33050
```


----------



## Sebastian (Feb 20, 2019)

What happens if you create a vimage test jail and attach it to vm-vlan33 ( vlan33 ) . I think vm-vlan33 is a renamed bridge ?


----------



## bautsche (Feb 21, 2019)

Right. I've now re-installed the system and installed it without XEN to make sure it doesn't interfere. I have also avoided creating any bridges during the automated install and instead installed the system over re0.
I have then logged on on the console, did an `ifconfig re0 inet 0.0.0.0`
I understand that re0 cannot be unplumbed, apparently that is expected behaviour and trying to do so causes:

```
ifconfig: SIOCIFDESTROY: invalid argument
```
OK, I have then plumbed up two separate interfaces: re0.33 and re0.40 with equivalent commands as follows:

```
ifconfig re0.33 plumb vlan 33 vlandev re0
ifconfig re0.33 inet 192.168.33.20 up
```
both of those interfaces now work, they can be pinged and they can ping their defaultrouter on their respective networks.

OK, time to take down re0.33:

```
ifconfig re0.33 down
ifconfig re0.33 unplumb
```
So, trying to get this interface configured as a bridge:

```
ifconfig bridge create
ifconfig bridge0 inet 192.168.33.20 addm re0.33
```
And it doesn't ping unfortunately.

So what is it I'm doing wrong?

Here's what my `netstat -rn` and my `ifconfig -a` looks like:

```
root@bianca # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/8          link#1             U           re0
127.0.0.1          link#2             UH          lo0
192.168.33.0/24    link#5             U       bridge0
192.168.33.20      link#5             UHS         lo0
192.168.40.0/24    link#3             U        re0.40
192.168.40.33      link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
root@bianca # ifconfig -a
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 80:ee:73:63:5c:48
        inet 0.0.0.0 netmask 0xff000000 broadcast 0.255.255.255 
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re0.40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 80:ee:73:63:5c:48
        inet 192.168.40.33 netmask 0xffffff00 broadcast 192.168.40.255 
        groups: vlan 
        vlan: 40 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0.33: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 80:ee:73:63:5c:48
        inet 0.0.0.0 netmask 0xff000000 broadcast 0.255.255.255 
        groups: vlan 
        vlan: 33 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:75:69:8e:39:00
        inet 192.168.33.20 netmask 0xffffff00 broadcast 192.168.33.255 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0.33 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        groups: bridge 
        nd6 options=9<PERFORMNUD,IFDISABLED>
root@bianca # ping -n 192.168.40.1
PING 192.168.40.1 (192.168.40.1): 56 data bytes
64 bytes from 192.168.40.1: icmp_seq=0 ttl=255 time=2.145 ms
64 bytes from 192.168.40.1: icmp_seq=1 ttl=255 time=18.048 ms
64 bytes from 192.168.40.1: icmp_seq=2 ttl=255 time=2.114 ms
^C
--- 192.168.40.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.114/7.436/18.048/7.504 ms
root@bianca # ping -n 192.168.33.1
PING 192.168.33.1 (192.168.33.1): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 192.168.33.1 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss
root@bianca #
```


----------

