# vlans over lagg



## vbelousov (Oct 12, 2009)

Hello.

I have perfectly working lagg0 interface and it was fine and ok until I've tried to put vlans over this lagg0.

Here I have following hardware and configuration:

*Cisco 2960G with:*

```
interface Port-channel4
 description NasBox_FEC
 switchport trunk native vlan 10
 switchport trunk allowed vlan 40,1001
 switchport mode trunk

interface GigabitEthernet0/13
 description NASBOX_em0
 switchport trunk native vlan 10
 switchport trunk allowed vlan 40,1001
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 4 mode active
!
interface GigabitEthernet0/14
 description NASBOX_em1
 switchport trunk native vlan 10
 switchport trunk allowed vlan 40,1001
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 4 mode active
```
*And FreeBSD 8RC1 with configured interface in rc.conf:*

```
cloned_interfaces="lagg0 vlan40 vlan1001"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 192.168.10.10 netmask 255.255.255.0"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_vlan40="inet 192.168.40.10 netmask 255.255.255.0 vlan 40 vlandev lagg0"
ifconfig_vlan1001="inet 192.168.20.10 netmask 255.255.255.0 vlan 1001 vlandev lagg0"
```
My problem is vlan interfaces. As well as native works fine and I can with no problem at all work with lagg0 thru native vlan as badly I cannot access tagged vlans Vlan40 and Vlan1001 interfaces.

Does any one know what I'm doing wrong?


----------



## Alt (Oct 12, 2009)

Maybe you need to put "switchport trunk encapsulation dot1q" in cat config


----------



## honk (Oct 12, 2009)

Please post output of "ifconfig -a" (in code tags).


----------



## vbelousov (Oct 12, 2009)

If've changed settings a bit. Here's what I have for now:


```
nasbox# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:02:a5:4e:9c:92
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:02:a5:4e:9c:92
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:02:a5:4e:9c:92
        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect
        status: active
        laggproto lacp
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:02:a5:4e:9c:92
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect
        status: active
        vlan: 10 parent interface: lagg0
vlan40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:02:a5:4e:9c:92
        inet 192.168.40.10 netmask 0xffffff00 broadcast 192.168.40.255
        media: Ethernet autoselect
        status: active
        vlan: 40 parent interface: lagg0
vlan1001: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:02:a5:4e:9c:92
        inet 192.168.20.10 netmask 0xffffff00 broadcast 192.168.20.255
        media: Ethernet autoselect
        status: active
        vlan: 1001 parent interface: lagg0
```

And for 2960

```
interface Port-channel4
 description NasBox_FEC
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,40,1001
 switchport mode trunk
!
interface GigabitEthernet0/13
 description NASBOX_em0
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,40,1001
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 4 mode active
!
interface GigabitEthernet0/14
 description NASBOX_em1
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,40,1001
 switchport mode trunk
 speed 1000
 duplex full
 channel-protocol lacp
 channel-group 4 mode active
```


----------



## vbelousov (Oct 12, 2009)

Alt said:
			
		

> Maybe you need to put "switchport trunk encapsulation dot1q" in cat config



Unfortunatelly here is no such option available in 2960.
Exact hardware is cisco WS-C2960G-24TC-L (PowerPC405) and software is 12.2(52)SE.

But it's could be set on router's switchport (2811 in my case). But it's useless.


----------



## honk (Oct 12, 2009)

Having the network 192.168.10.0/24 on both, native on lagg0 and tagged on vlan10 seems weird. What exactly is not working? Have you tried to ping your FreeBSD box from a different system in vlan40 or vlan1001 while watching with tcpdump on lagg0 to see if packets come in (and correctly tagged by the Cisco)? E.g. tcpdump -i lagg0 -e -n


----------



## vbelousov (Oct 13, 2009)

honk said:
			
		

> Having the network 192.168.10.0/24 on both, native on lagg0 and tagged on vlan10 seems weird. What exactly is not working? Have you tried to ping your FreeBSD box from a different system in vlan40 or vlan1001 while watching with tcpdump on lagg0 to see if packets come in (and correctly tagged by the Cisco)? E.g. tcpdump -i lagg0 -e -n



Yeah, it's kinda weird but technically it is possible on basic conception. And it shouldn't be a problem at all. Anyway I've tried both way with no changes. So just imagine that there is no Vlan10 interface. 

Yes, I had a few tries yesterday while I had long relationships with these pieces. I tried to ping it from switch and router and backwardly from BSD I pinged switch and router. Either way no positive results for all Vlan interfaces. And stable ping for native lagg one. My thoughts it that probably this configuration isn't possible for BSD (it's strange, because a was able to do even on Windows Server thru NIC drivers) at all??? I hadn't similar experience before.

Okay, I'll get dumps later when will get back to home.


----------



## honk (Oct 13, 2009)

It's definitely possible to use vlan tagging on lagg, I used it some time ago in 7.2, also with the em driver (Intel PRO/1000 MT, PT and GT cards). I just had some trouble with the configuration using rc.conf. Configuration by hand (ifconfig bla) was possible without any problems, but storing this configuration in rc.conf was a little bit confusing, at the end it worked.


----------



## vbelousov (Oct 13, 2009)

honk said:
			
		

> It's definitely possible to use vlan tagging on lagg, I used it some time ago in 7.2, also with the em driver (Intel PRO/1000 MT, PT and GT cards).



Yep, that's exactly what I expect to hear but had no proofs from manuals or google.



			
				honk said:
			
		

> I just had some trouble with the configuration using rc.conf. Configuration by hand (ifconfig bla) was possible without any problems, but storing this configuration in rc.conf was a little bit confusing, at the end it worked.



Okay then. So would you be so kind to publish part of your rc.conf of course if its available for now?

I'm still on work and can't get console to home server. :^(


----------



## vbelousov (Oct 13, 2009)

It wasn't good ideat to dump on lagg0... In final it became as 8+ MB file for 10 seconds dump.
I did it for vlan40 only which was under Cat ping in this time. So I'll just put it here in code tag.


```
20:57:09.402751 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:11.403640 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:12.479471 00:0a:b8:2c:b6:51 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.26 tell 192.168.40.1, 
length 46
20:57:13.309154 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:13.309161 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:13.408522 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:15.320784 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:15.320791 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:15.413406 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:17.333911 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:17.333919 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:17.418289 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:19.347166 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:19.347173 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:19.427295 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:21.360545 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:21.360553 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:21.427932 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:23.373673 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:23.373680 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:23.432816 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:25.386926 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:25.386934 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:25.437827 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:27.400306 00:1d:e5:48:13:c2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.40.10 tell 192.168.40.250, 
length 46
20:57:27.400313 00:02:a5:4e:9c:92 > 00:1d:e5:48:13:c2, ethertype ARP (0x0806), length 42: Reply 192.168.40.10 is-at 00:02:a5:4e:9c:92, length 
28
20:57:27.442583 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
20:57:29.450715 00:1d:e5:48:13:8e > 01:00:0c:cc:cc:cd, 802.3, length 64: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 
0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1d, Config, Flags [none], bridge-id 8028.00:1d:e5:48:13:80.8050, length 42
```

Unfortunately I can't attach big dump but can send it thru email.

Based on information from tcpdump I can see that at least who-has requests are successfuly reaches to vlan40 interface. And even it goes back to Catalyst (which has no information in arp table regading vlan40 interface). %(

But! Here's what I've got for other vlans (Vlan10 and Vlan1001) at 2960 arp's table:

```
Internet  192.168.10.11           0   Incomplete      ARPA
Internet  192.168.20.10           0   Incomplete      ARPA
```

For now it looks even stranger...


----------



## honk (Oct 14, 2009)

I'm confused about your setup... I understood that your FreeBSD box has a trunk to a Cisco 2960. This is a layer-2 switch, therefore you can't have more than one ip-address at the same time on the 2960.

Who is trying to ping the vlan40 ip of your FreeBSD box in the capture above? Who has 192.168.40.250? Your 2960 on Interface Vlan40? If so it's clear that you have an incomplete arp-cache for vlan10 and vlan1001 on the 2960.

Could you please extract only arp-traffic from the big capture file and post it here, I assume you did not capture 8MBytes of arp traffic, but this should clarify if the FreeBSD box sent the arp replies with the proper vlan tag.

*tcpdump -n -e -r yourcapturefile.cap arp*

At the moment I have no access to the box I configured with lagg and vlan recently, so I can't post the rc.conf...

cheers,
honk


----------



## vbelousov (Oct 14, 2009)

honk said:
			
		

> I'm confused about your setup... I understood that your FreeBSD box has a trunk to a Cisco 2960. This is a layer-2 switch, therefore you can't have more than one ip-address at the same time on the 2960.



Why not? I can setup one or few IP addresses on Vlan interfaces and this is what I actually have right now (Vlan10 192.168.10.250 - 'secure network', Vlan40 192.168.40.250 - 'unsecure network' and Vlan1001 - 'voice'). And L3-ports (no switchport) also able to has one or few IPs.

And then I configured different vlans on NAS with different NFS exports for each vlan. This was my point when I designed this solution.



			
				honk said:
			
		

> Who is trying to ping the vlan40 ip of your FreeBSD box in the capture above? Who has 192.168.40.250? Your 2960 on Interface Vlan40? If so it's clear that you have an incomplete arp-cache for vlan10 and vlan1001 on the 2960.



Yes, I've tried to ping NASBOX's vlan40 interface from 2960 vlan40 interface.
What is the reason which cause this problem with arp-cache. Unfortunately it's not clear for me. Would you please be so kind to explain to me?



			
				honk said:
			
		

> Could you please extract only arp-traffic from the big capture file and post it here, I assume you did not capture 8MBytes of arp traffic, but this should clarify if the FreeBSD box sent the arp replies with the proper vlan tag.



*tcpdump -n -e -r yourcapturefile.cap arp*[/QUOTE]

Alright, I also will shut down whole other network related service to decrease ability of appearance of noise information.
Will post it in 10 or 12 hours.



			
				honk said:
			
		

> At the moment I have no access to the box I configured with lagg and vlan recently, so I can't post the rc.conf...



It's okay, thank you anyway for you attention to my issue and for you help.

Vitaly.


----------



## vbelousov (Oct 18, 2009)

I took timeout. Just got another NIC for one of vlans. Will get back to this configuration later.


----------



## Barma (Nov 17, 2009)

*the same problem*

1) Cisco C3750-ADVIPSERVICESK9-M
2) FreeBSD vpn1 8.0-RC3 FreeBSD 8.0-RC3 #2: Fri Nov 13 16:24:02 UTC 2009

Cisco config:

```
interface Port-channel2
 description sv_VPN_2_NAS
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 29
 switchport trunk allowed vlan 29,96
 switchport mode trunk
 switchport nonegotiate
!
interface GigabitEthernet1/0/8
 description sv_vpn2_em0
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 29
 switchport trunk allowed vlan 29,96
 switchport mode trunk
 switchport nonegotiate
 no cdp enable
 channel-protocol lacp
 channel-group 2 mode active
!
interface GigabitEthernet1/0/9
 description sv_vpn2_em1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 29
 switchport trunk allowed vlan 29,96
 switchport mode trunk
 switchport nonegotiate
 no cdp enable
 channel-protocol lacp
 channel-group 2 mode active
!
interface Vlan96
 ip address 1.1.1.1 255.255.255.0
!
```

BSD config:

```
defaultrouter="x.x.x.1"
gateway_enable="yes"

cloned_interfaces="lagg0 vlan96"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_lagg0="up laggproto lacp laggport em0 laggport em1 x.x.x.2 netmask 255.255.255.248"
ifconfig_vlan96="vlan 96 vlandev lagg0 1.1.1.2/24"
```


```
ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:16:76:ad:75:52
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:16:76:ad:75:52
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:16:76:ad:75:52
        inet x.x.x.2 netmask 0xfffffff8 broadcast x.x.x.3
        media: Ethernet autoselect
        status: active
        laggproto lacp
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan96: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:16:76:ad:75:52
        inet 1.1.1.2 netmask 0xffffff00 broadcast 1.1.1.255
        media: Ethernet autoselect
        status: active
        vlan: 96 parent interface: lagg0
```
Native vlan 29 works fine. Vlan iface is not work.

I can't ping from 1.1.1.2 to 1.1.1.1

```
tcpdump -nvi lagg0 vlan 96

12:00:37.230313 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
12:00:38.266345 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
12:00:39.302389 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
12:00:40.338424 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
12:00:41.374664 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
```
Cisco can't see the ARP!

tcpdump -nvi lagg0 vlan 96 for ping from cisco to bsd (from 1.1.1.1 to 1.1.1.2):

```
11:58:23.243114 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.2 tell 1.1.1.1, length 46
11:58:23.243132 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.1.1.2 is-at 00:16:76:ad:75:52, length 28
11:58:25.240463 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.2 tell 1.1.1.1, length 46
11:58:25.240471 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.1.1.2 is-at 00:16:76:ad:75:52, length 28
11:58:27.245428 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.2 tell 1.1.1.1, length 46
11:58:27.245437 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.1.1.2 is-at 00:16:76:ad:75:52, length 28
11:58:29.250396 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.2 tell 1.1.1.1, length 46
11:58:29.250403 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.1.1.2 is-at 00:16:76:ad:75:52, length 28
```

But BSD box see the ARP!

```
arp -an
? (1.1.1.1) at 00:17:e0:58:d3:47 on vlan96 [vlan]
? (1.1.1.2) at 00:16:76:ad:75:52 on vlan96 permanent [vlan]
```

now ping from 1.1.1.2 to 1.1.1.1 again

```
12:02:42.589389 IP (tos 0x0, ttl 64, id 12673, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.2 > 1.1.1.1: ICMP echo request, id 50693, seq 131, length 64
12:02:43.625424 IP (tos 0x0, ttl 64, id 12681, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.2 > 1.1.1.1: ICMP echo request, id 50693, seq 132, length 64
12:02:44.661981 IP (tos 0x0, ttl 64, id 12684, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.2 > 1.1.1.1: ICMP echo request, id 50693, seq 133, length 64
12:02:45.698006 IP (tos 0x0, ttl 64, id 12687, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.2 > 1.1.1.1: ICMP echo request, id 50693, seq 134, length 64
```

Cisco cant get traffic from vlan 96 at all...


----------



## vbelousov (Nov 18, 2009)

It seems kindly you have absolutely the same situation that I described in beginning. I have a guess that it could have two cases. First one is - em drivers does not support vlans over lagg. Second one is - lagg implementation doesn't support vlans over itself. Second case is not so really convincing because of response from honk. He told that he has properly working environment...


----------



## vbelousov (Nov 18, 2009)

Has anyone the same environment (vlans over lagg) with different NICs?


----------



## LMik (Nov 18, 2009)

vbelousov said:
			
		

> Has anyone the same environment (vlans over lagg) with different NICs?



Having same problem on em+lagg+vlan on 

FreeBSD pppoe3.vosnet.ru 8.0-PRERELEASE FreeBSD 8.0-PRERELEASE #0: Mon Nov 16 20

But in 7.2 in works fine.


----------



## Orum (Nov 18, 2009)

vbelousov said:
			
		

> Has anyone the same environment (vlans over lagg) with different NICs?



After a bit of a struggle (with lagg itself, not vlans), I have vlans working over lagg (including the native vlan on the lagg interface), with two vr nics.  Seems like this issue is probably specific to something with either the cisco hardware/firmware (I'm connected to an HP switch via FEC) or, more likely, em + lagg + vlans.


----------



## Barma (Nov 18, 2009)

FreeBSD 8.0 RC3 (em + lagg(lacp) + vlan) not working with Qtech 3900
more likely em + lagg + vlans problem in 8.0, not cisco hardware...


----------



## vbelousov (Nov 19, 2009)

Huh, situation does not became more clear for me. 
On the one hand we have a problem with 'em' on FreeBSD 8. But it's not a problem to run vlans over 'em's lagg on 7.2. Probable it is happen because of older version of em drivers on 7.2. On the other hand we have perfectly working environment with vr NICs. For 8RC1 I downloaded latest version from intel.com and finally I have what I have...

As far as we haven't any one who has successfully configured 'em' lagg for vlans, I will take the view it's kindly a problem of 'em' interface drivers. Probably we could report a bug to Intel networking division.


----------



## Orum (Nov 19, 2009)

vbelousov said:
			
		

> On the other hand we have perfectly working environment with vr NICs.


Yes, but I'm also on 7.2.  I haven't tried this on any of the 8.0 RCs, so for all I know I may have issues with vr + lagg + vlans + 8.0.

If I have time I'll test this on 8.0-RC3, but it's a low priority for me right now, sorry :\.


----------



## DutchDaemon (Nov 19, 2009)

Barma said:
			
		

> FreeBSD 8.0 RC3 (em + lagg(lacp) + vlan) not working with Qtech 3900
> more likely em + lagg + vlans problem in 8.0, not cisco hardware...



As posted in another thread:
http://www.freebsd.org/cgi/query-pr.cgi?pr=132715


----------



## vbelousov (Nov 19, 2009)

Aha... So finally I can conclude that the problem is in 'em' drivers. And 8RCx contains newer version which is not working properly with vlans over lagg interface. Or even with lagg interface at all in some conditions.

I guess I have to try that patch.
DutchDaemon - thanks.


----------



## Barma (Nov 20, 2009)

I have not kernel panic... and cant check that patch...
this server is working now (one iface - external, second iface - internal+vlans)
I can try this patch on another server, but not now. Dec or Jan maybe.


----------



## knarf (Nov 24, 2009)

My upgrade path was like this:

FreeBSD 7.2, em0/em1, lagg0: vlans worked fine (Netgear Switch)

Upgraded to 8.0: Network does not work. arp -a shows the other MAC addresses, but ping and everything else does not work.

I've read in the em-Manual:

BUGS
     Hardware-assisted VLAN processing is disabled by default.  You can enable
     it on an em interface using ifconfig(8).

So I tried ifconfig_emN="vlanmtu vlanhwtag vlanhwfilter up". This did not change anything. Later I saw this "bug" was already present in 7.2. But it worked fine in 7.2.

My solution was to change the NICs from em to re (sigh!).


```
ifconfig_re0="up"
ifconfig_re1="up"

cloned_interfaces="lagg0 vlan0 vlan1 vlan3 vlan4"

ifconfig_lagg0="laggproto lacp laggport re0 laggport re1"
ifconfig_vlan0="inet 172.17.16.254/24 vlan 3 vlandev lagg0"
ifconfig_vlan1="inet 172.17.4.254/24  vlan 5 vlandev lagg0"
ifconfig_vlan3="inet 172.17.2.254/24 vlan 2 vlandev lagg0"
ifconfig_vlan3_alias0="inet 172.17.2.2/32"
ifconfig_vlan4="inet 10.0.0.1/24 vlan 4 vlandev lagg0"
```


----------



## vbelousov (Nov 25, 2009)

Huh, funny...

Thanks for your info Knarf.

By the way. I've found that Intel placed new version of em drivers. 

em-6.9.21.tar.gz	Ver:6.9.21	Date:11/23/2009	Size:199 (KB)

Will try them in a short time...


----------



## vbelousov (Nov 30, 2009)

Nope, it doesn't help.
I guess that I will take one more NIC for vlans. 
Welcome Realtek, because Intel can't make a proper driver?


----------



## knarf (Nov 30, 2009)

Realtek is also having problems: http://forums.freebsd.org/showthread.php?t=7193

And my local dealer has nothing but Intel or Realtek.


----------



## vbelousov (Nov 30, 2009)

I was joking a bit. 
Two marvels (sk and msk) are integrated in motherboard that I have.


----------



## jbhappy (Dec 16, 2009)

To anyone who is having non-panic trouble with em+lagg+vlan: I think I've identified the problem, as I ran into it myself. tcpdumping on the FreeBSD box didn't really show it off, but once I put something on the other end of the active cable to catch frames, I saw that the FreeBSD box is transmitting what wireshark identifies as ISL-tagged frames instead of 802.1q-tagged frames. I've filed PR kern/141646 in the hopes it can get addressed before I need to upgrade my fleet of 7.1 PowerEdge 2850s to 8.x.


----------



## vbelousov (Dec 17, 2009)

Nice job Jeff.
Thanks a lot.


----------



## m1m1n0 (Jan 29, 2010)

*A workaround*

Hello!

The is a workaround for the problem.


```
ifconfig_em0="-vlanhwtag up"
ifconfig_em1="-vlanhwtag up"
ifconfig_em2="-vlanhwtag up"
ifconfig_em3="-vlanhwtag up"
cloned_interfaces="lagg0 vlan10"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 laggport em2 laggport em3 up"
ifconfig_vlan10="vlan 10 vlandev lagg0 192.168.10.2/24"
```

Cheers!


----------



## vbelousov (Jan 30, 2010)

Nope, doesn't work for me.


----------



## vermaden (Jan 30, 2010)

@vbelousov

I remember that people reported, that *em* driver has a bug that prevents this (vlan over lagg) because if they moved to Realtek cards (with re driver), vlan over lagg worked like a charm, check mailing lists/ask developers/submit a bug.


----------



## m1m1n0 (Jan 30, 2010)

*It is in fact working very well*

Ok, let me add more information.

Stock kernel of 8.0-STABLE-201001 (snapshot) had the problem in question. The latest available Intel driver (6.9.21) has the problem also.

In my case, removal of hardware assisted VLAN tagging and hardware assisted VLAN filtering made it working. I tog ~10 perfectly working VLANS at the moment, right how it was in 7.2-STABLE. I discovered that switching off VLAN tagging alone is enough. However, in addition to that I keep TCP Segmentation Offload switched off (and my complete configuration for an interface is:

```
ifconfig_em0="rxcsum txcsum -tso -lro promisc polling -vlanhwtag up"   # Keep TSO disabled!!!
```
and I do not remember why I have the commentary like that, but I had very strong reasons back then to do so).


I will spend some time going through the source code. It really feels like the packets are tagged twice when they go out of the card...

Cheers!


----------



## vbelousov (Jan 30, 2010)

vermaden said:
			
		

> @vbelousov
> 
> I remember that people reported, that *em* driver has a bug that prevents this (vlan over lagg) because if they moved to Realtek cards (with re driver), vlan over lagg worked like a charm, check mailing lists/ask developers/submit a bug.



Yeah, I do remember that too. But I decided to stay on Intel. At lease for a while. Basicly because of multy port configuragtion. I have one quaded and one dual PT and MT controllers. So it's definetely not an option for me. But I have to thank you for your participation pal.


----------



## vermaden (Jan 31, 2010)

@vbelousov

You are welcome mate.


----------



## dam23 (Feb 1, 2010)

m1m1n0 said:
			
		

> ```
> ifconfig_em0="rxcsum txcsum -tso -lro promisc polling -vlanhwtag up"   # Keep TSO disabled!!!
> ```




Works just fine here, thanks a lot for sharing the workaround.
This is saving me the troublesome and very dirty solution of not tagging VLANs anymore and splitting my interfaces, 1 per access  vlan on the switch, without redundancy.


----------



## vbelousov (Feb 1, 2010)

dam23 said:
			
		

> Works just fine here, thanks a lot for sharing the workaround.
> This is saving me the troublesome and very dirty solution of not tagging VLANs anymore and splitting my interfaces, 1 per access  vlan on the switch, without redundancy.



Dear dam23, what NICs do you have exactly? And what FreeDSB version?


----------



## m1m1n0 (Feb 1, 2010)

I won't say for dam23, my system is:

[CMD="FreeBSD xx.yyyy.com 8.0-STABLE FreeBSD 8.0-STABLE #3: Tue Jan 26 23:30:00 EET 2010     [email]igor@xx.yyyy.com[/email]:/usr/obj/usr/src/sys/FSFWCLUSTER  amd64"][/CMD]
and the em driver is loaded as a module compiled from the source code on the Intel web site.


```
em0: <Intel(R) PRO/1000 Network Connection 6.9.21> port 0xdce0-0xdcff mem 0xfd9e0000-0xfd9fffff,0xfd9c0000-0xfd9dffff irq 16 at device 0.0 on pci10
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:15:17:ff:d8:1e

---------------

em0@pci0:10:0:0:        class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'HP NC360T PCIe DP Gigabit Server Adapter (n1e5132)'
    class      = network
    subclass   = ethernet

---------------

Intel PRO/1000 PT Dual Port Server Adapter (82571)
```

funny that it shows HP NC360T - the card came with a Dell server.


----------



## m1m1n0 (Feb 1, 2010)

```
FreeBSD xx.yyyy.com 8.0-STABLE FreeBSD 8.0-STABLE #3: Tue Jan 26 23:30:00 EET 2010     igor@xx.yyyy.com:/usr/obj/usr/src/sys/FSFWCLUSTER  amd64
```


----------



## vbelousov (Feb 2, 2010)

Kinda strange that I had no effect by switching off hardware VLAN tagging. Does anyone also has not a positive result?
In any way I'll try it again tonight. May be I did something wrong.
And thank you guys.


----------



## dam23 (Feb 2, 2010)

@vbelousov: for some reason and apparently independently of "spanning-tree portfast trunk" on the switch, it takes the box around 30 seconds after becoming fully up (daemons started + console login available) for it to get network connectivity.

Might have to do with us using aggregation, I don't know.

Perhaps you've got the same issue ?
Try giving the box a little time after booting, see how that works out.


The configuration we're running here is:

```
FreeBSD 8.0-RELEASE-p2 #0: Tue Jan  5 21:11:58 UTC 2010
```
Upgraded from 7.2 via freebsd-update
GENERIC kernel for now
amd64


```
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0xdce0-0xdcff mem 0xfc3e0000-0xfc3fffff,0xfc3c0000-0xfc3dffff irq 16 at device 0.0 on pci12

em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0xdcc0-0xdcdf mem 0xfc3a0000-0xfc3bffff,0xfc380000-0xfc39ffff irq 17 at device 0.1 on pci12

em0@pci0:12:0:0:	class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00

em1@pci0:12:0:1:	class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
```

Intel cards aggregated as a lagg0 device:

```
ifconfig_em0="-vlanhwtag -lro -tso up"
ifconfig_em1="-vlanhwtag -lro -tso up"
ifconfig_lagg0="laggproto failover laggport em0 laggport em1"
ifconfig_vlan16="inet [snip] vlan 16 vlandev lagg0"
```


----------



## vbelousov (Feb 6, 2010)

dam23

You have not created interfaces by cloned_interface command. Your config shouldn't work.

To all.
Thank you all guys, I've tried these keys again and for now it works fine. As dam23 said I just took a little break and it got up.


----------



## dam23 (Feb 8, 2010)

Actually, I do have the following cloned interfaces:

```
cloned_interfaces="lagg0 lagg1 vlan666 vlan16 vlan26 vlan36 carp0 carp1 carp2"
```

I didn't think they were all that relevant so I omitted this line.


Glad it works for you too now.
This solution should be put somewhere in the release notes or wherever.


----------



## jbhappy (Feb 8, 2010)

A patch for this issue has been developed by Ermal LuÃ§i so that disabling hardware VLAN tagging should no longer be necessary. It's in HEAD now and will be in 8-STABLE in a couple weeks and then in 8.1-RELEASE. See http://lists.freebsd.org/pipermail/freebsd-net/2010-February/024477.html.


----------



## knarf (Aug 21, 2010)

jbhappy said:
			
		

> A patch for this issue has been developed by Ermal LuÃ§i so that disabling hardware VLAN tagging should no longer be necessary. It's in HEAD now and will be in 8-STABLE in a couple weeks and then in 8.1-RELEASE. See http://lists.freebsd.org/pipermail/freebsd-net/2010-February/024477.html.



The problem still exists in 8.1-RELEASE.

The patch is only in HEAD (r203548) but not in stable. 

http://redmine.pfsense.org/reposito...?rev=64fd5b02c883db52511a333d8b879eaad2038d11

I've tried it right now and it works (vlan->lagg->em).


----------



## grabes (Dec 14, 2010)

Anyone have any idea, why this is not making it into stable??  Is there stability issues?


----------



## phoenix (Dec 15, 2010)

Contacting the author of the patch may give some insight.  Not many devs frequent the forums.

Or, maybe sending an e-mail to the freebsd-current and/or freebsd-stable mailing lists.


----------

