# Weird LAGG behavior



## dougs (Mar 25, 2014)

Hello-

I've set up a LAGG connection between a FreeBSD 9.2-RELEASE-p3 machine and a Cisco SG300-52 switch. Indications from both sides report that the LAGG connection has been made and is functioning. However, I am seeing network speeds that are much slower than when I had a single NIC connection.

I do daily overnight backups using different backup protocols. One robocopy backup via smbd that usually takes about 3 hours now takes about 5 hours to complete over the network. Another remote backup that uses rsync also takes more time (almost twice as long) to complete when using LAGG.

Here's the network hardware configuration:

```
root@test:/root # dmesg | g em0
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xf8900000-0xf891ffff irq 17 at device 0.0 on pci3
em0: Using an MSI interrupt
em0: Ethernet address: 00:13:20:b0:48:1d
root@test:/root # dmesg | g em1
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x1200-0x123f mem 0xf8820000-0xf883ffff,0xf8800000-0xf881ffff irq 18 at device 2.0 on pci4
em1: Ethernet address: 90:e2:ba:15:9a:ba
[12]root@test:/root #
```

I'm wondering of the issue is due to different NIC hardware? The first one is built into the motherboard while the other is a PCI card.

My LAGG config:

```
# LAGG configuratiton
ifconfig_em0="up"
ifconfig_em1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 192.168.xxx.xx/24"
```


```
root@test:/root # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:13:20:b0:48:1d
        inet6 fe80::213:20ff:feb0:481d%em0 prefixlen 64 scopeid 0x1 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:13:20:b0:48:1d
        inet6 fe80::92e2:baff:fe15:9aba%em1 prefixlen 64 scopeid 0x7 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:13:20:b0:48:1d
        inet 192.168.xxx.xx netmask 0xffffff00 broadcast 192.168.xxx.255
        inet6 fe80::213:20ff:feb0:481d%lagg0 prefixlen 64 scopeid 0x9 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
root@test:/root #
```

I see that em0 contains two additional options that em1 doesn't have. TS04 and VLAN_HWTSO.


```
root@test:/root # cat /etc/sysctl.conf
# $FreeBSD: releng/9.1/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0

# tunable zfs parameters
kern.maxvnodes=250000
#vfs.zfs.write_limit_override=268435456
#kern.maxfiles=16384
kern.maxfiles=204800
#kern.maxfilesperproc=16384
kern.maxfilesperproc=200000
#kern.ipc.maxsockbuf=2097152        # kernel socket buffer space
kern.ipc.nmbclusters=262144       # kernel mbuf space raised 275MB of kernel dedicated ram
kern.ipc.somaxconn=4096          # size of the listen queue for accepting new TCP connections
kern.ipc.maxsockets=204800        # increase the limit of the open sockets


# added 12/16/09 to improve system performance when writing millions of files
#vfs.read_max=256
#vfs.hirunningspace=2097152

# the following sysctl variables were suggested by this web site at
# Calomel.org. https://calomel.org/network_performance.html
#
#kern.ipc.maxsockbuf=2097152        # kernel socket buffer space
#kern.ipc.nmbclusters=262144       # kernel mbuf space raised 275MB of kernel dedicated ram
#kern.ipc.somaxconn=32768          # size of the listen queue for accepting new TCP connections
#kern.ipc.maxsockets=204800        # increase the limit of the open sockets
# kern.randompid=348                # randomized processes id's
# net.inet.icmp.icmplim=50          # reply to no more than 50 ICMP packets per sec
# net.inet.ip.process_options=0     # do not processes any TCP options in the TCP headers
# net.inet.ip.redirect=0            # do not allow ip header redirects
# net.inet.ip.rtexpire=2            # route cache expire in two seconds
# net.inet.ip.rtminexpire=2         # "
#net.inet.ip.rtmaxcache=256        # route cache entries increased
# net.inet.icmp.drop_redirect=1     # drop icmp redirects
net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports
# net.inet.tcp.delayed_ack=0        # no need to delay ACK's
# net.inet.tcp.drop_synfin=1        # drop TCP packets which have SYN and FIN set
# net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 seconds (default 30)
# net.inet.tcp.nolocaltimewait=1    # do not create TIME_WAIT state for localhost
# net.inet.tcp.path_mtu_discovery=0 # disable MTU path discovery
#net.inet.tcp.recvbuf_max=2097152   # TCP receive buffer space
#net.inet.tcp.recvbuf_max=16777216   # TCP receive buffer space
#net.inet.tcp.recvspace=8192       # decrease buffers for incoming data
#net.inet.tcp.sendbuf_max=16777216   # TCP send buffer space
#net.inet.tcp.sendbuf_max=2097152   # TCP send buffer space
#net.inet.tcp.sendspace=16384      # decrease buffers for outgoing data
net.inet.udp.blackhole=1          # drop any UDP packets to closed ports
# security.bsd.see_other_uids=0     # keeps users segregated to their own processes list
# security.bsd.see_other_gids=0     # "

#net.isr.direct=1
#net.isr.direct_force=1
#dev.em.0.rx_int_delay=200
#dev.em.0.tx_int_delay=200
#dev.em.0.rx_abs_int_delay=4000
#dev.em.0.tx_abs_int_delay=4000
#dev.em.0.rx_processing_limit=4096
##dev.em.0.rx_processing_limit=-1
#dev.em.1.rx_int_delay=200
#dev.em.1.tx_int_delay=200
#dev.em.1.rx_abs_int_delay=4000
#dev.em.1.tx_abs_int_delay=4000
#dev.em.1.rx_processing_limit=4096
##dev.em.1.rx_processing_limit=-1
root@test:/root #
```

I've played with the options in /etc/sysctl.conf but am unable to get a clean working LAGG connection. On another FreeBSD server using two embedded NIC motherboard connections in a LAGG configuration, I'm seeing excellent networking conditions.

Admittedly, this issue is on a much older machine- one that is roughly 8 years old. It functions quite well for its age but I would be surprised if it's due to age.

Are the NICs too dissimilar hardware-wise to function well as a LAGG pair?

~Doug


----------



## junovitch@ (Mar 25, 2014)

I'm not a LAGG expert, but I am curious what the Cisco side reports.  Can you show the output of, say, `show etherchannel summary` and a `show interface` for both physical interfaces and the PO interface on the switch?


----------



## dougs (Mar 26, 2014)

I'm no Cisco expert either. In fact today was the first time I had established a ssh session to the switch.

`show etherchannel summary` doesn't exist. We're running the latest firmware (1.3.5.58) on this switch and it appears etherchannel isn't supported.

`show interface` wasn't adequate, it asked for an additional parameter. So I went with this:


```
switch0f5e86#show interfaces status
                                             Flow Link          Back   Mdix
Port     Type         Duplex  Speed Neg      ctrl State       Pressure Mode
-------- ------------ ------  ----- -------- ---- ----------- -------- -------
gi1      1G-Copper      --      --     --     --  Down           --     --    
gi2      1G-Copper    Full    100   Enabled  Off  Up          Disabled On     
gi3      1G-Copper      --      --     --     --  Down           --     --    
gi4      1G-Copper      --      --     --     --  Down           --     --    
gi5      1G-Copper      --      --     --     --  Down           --     --    
gi6      1G-Copper    Full    1000  Enabled  Off  Up          Disabled On     
gi7      1G-Copper      --      --     --     --  Down           --     --    
gi8      1G-Copper      --      --     --     --  Down           --     --    
gi9      1G-Copper      --      --     --     --  Down           --     --    
gi10     1G-Copper      --      --     --     --  Down           --     --    
gi11     1G-Copper      --      --     --     --  Down           --     --    
gi12     1G-Copper    Full    1000  Enabled  On   Up          Disabled On     
gi13     1G-Copper    Full    100   Enabled  Off  Up          Disabled On     
gi14     1G-Copper    Full    100   Enabled  Off  Up          Disabled Off    
gi15     1G-Copper      --      --     --     --  Down           --     --    
gi16     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi17     1G-Copper      --      --     --     --  Down           --     --    
gi18     1G-Copper      --      --     --     --  Down           --     --    
gi19     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi20     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi21     1G-Copper      --      --     --     --  Down           --     --    
gi22     1G-Copper    Full    1000  Enabled  On   Up          Disabled On     
gi23     1G-Copper    Full    100   Enabled  Off  Up          Disabled On     
gi24     1G-Copper      --      --     --     --  Down           --     --    
gi25     1G-Copper      --      --     --     --  Down           --     --    
gi26     1G-Copper      --      --     --     --  Down           --     --    
gi27     1G-Copper      --      --     --     --  Down           --     --    
gi28     1G-Copper      --      --     --     --  Down           --     --    
gi29     1G-Copper      --      --     --     --  Down           --     --    
gi30     1G-Copper      --      --     --     --  Down           --     --    
gi31     1G-Copper      --      --     --     --  Down           --     --    
gi32     1G-Copper      --      --     --     --  Down           --     --    
gi33     1G-Copper      --      --     --     --  Down           --     --    
gi34     1G-Copper      --      --     --     --  Down           --     --    
gi35     1G-Copper      --      --     --     --  Down           --     --    
gi36     1G-Copper    Full    1000  Enabled  On   Up          Disabled On     
gi37     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi38     1G-Copper      --      --     --     --  Down           --     --    
gi39     1G-Copper    Full    100   Enabled  Off  Up          Disabled On     
gi40     1G-Copper      --      --     --     --  Down           --     --    
gi41     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi42     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi43     1G-Copper      --      --     --     --  Down           --     --    
gi44     1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off    
gi45     1G-Copper      --      --     --     --  Down           --     --    
gi46     1G-Copper    Full    1000  Enabled  On   Up          Disabled On     
gi47     1G-Copper    Full    100   Enabled  Off  Up          Disabled Off    
gi48     1G-Copper      --      --     --     --  Down           --     --    
gi49     1G-Copper      --      --     --     --  Down           --     --    
gi50     1G-Copper    Full    100   Enabled  Off  Up          Disabled On     
gi51     1G-Combo-C   Full    1000  Enabled  Off  Up          Disabled On     
gi52     1G-Combo-C   Full    1000  Enabled  Off  Up          Disabled On     

                                          Flow    Link        
Ch       Type    Duplex  Speed  Neg      control  State       
-------- ------- ------  -----  -------- -------  ----------- 
Po1      1G      Full    1000   Enabled  Off      Up          
Po2      1G      Full    1000   Enabled  On       Up          
Po3      1G      Full    1000   Enabled  On       Up          
Po4         --     --      --      --       --    Not Present 
Po5         --     --      --      --       --    Not Present 
Po6         --     --      --      --       --    Not Present 
Po7         --     --      --      --       --    Not Present 
switch0f5e86#
```

As a side note, I changed the flowcontrol option for the LAGG connection on the switch to Auto. Checking last night's backup, it appears to have speed up the backups. I think it's still too early to pass a verdict until more backups are performed- especially those large weekly backups. Unfortunately I will be out of the office until late next week. I will share once I am back.

Feel free to make recommendations as to how the switch can be best configured for maximum flow and efficiency.

~Doug


----------



## junovitch@ (Mar 26, 2014)

Sorry for the confusion, I'm a bit used to the IOS syntax and display output.  I downloaded the manual for that switch from here: http://www.cisco.com/c/dam/en/us/td.../administration_guide/CLI_Nikola300_1_3_7.pdf.  It looks like `show lacp` and `show interface counters` were what commands I was thinking about.  

What I'm looking for here is nothing crazy in the counters with regards errors and that LACP shows that the interface is actively bonded.  I've seen some odd behavior with etherchannels when one of the members has a link quality issues and the they are hard-coded to be on rather than doing an LACP negotiation as either active/active or active/passive.   You'll want to set `channel-group <channel #> mode auto` on the switch side so it can actively negotiate.  The issue I had seen showed itself as a performance hit for TCP traffic and major drop outs on some multicast UDP video streams.  TCP traffic was able to re-transmit to deal with the errors one of the physical interfaces had but when UDP traffic was going over that poor link our video quality went bad big time.


----------



## dougs (Mar 26, 2014)

It appears that the firmware wants additional parameters to `show lacp`:


```
switch0f5e86#show lacp
% missing mandatory parameter
switch0f5e86#show lacp 
  Port-Channel         IEEE 802.3 link aggregation information of port-channel
  GigabitEthernet      IEEE 802.3 Gigabit Ethernet port id
  GE                   IEEE 802.3 Gigabit Ethernet port id
  parameters           IEEE 802.3 link aggregation parameters information
  statistics           IEEE 802.3 link aggregation statistics information
  protocol-state       IEEE 802.3 link aggregation protocol-state information
switch0f5e86#
```

Which one were you looking for?


```
switch0f5e86#show interfaces counter

      Port       InUcastPkts  InMcastPkts  InBcastPkts    InOctets   
---------------- ------------ ------------ ------------ ------------ 
      gi1             0            0            0            0       
      gi2          3339582       15600         9979      2951451485  
      gi3             0            0            0            0       
      gi4             0            0            0            0       
      gi5             0            0            0            0       
      gi6           72877         8218         314        22981790   
      gi7             0            0            0            0       
      gi8             0            0            0            0       
      gi9             0            0            0            0       
      gi10            0            0            0            0       
      gi11            0            0            0            0       
      gi12        187001940       2869          24      249707055501 
      gi13          107423        102          2024       14186982   
      gi14          50237         169          310        7021609    
      gi15            0            0            0            0       
      gi16          74859         594          5927       9261914    
      gi17            0            0            0            0       
      gi18            0            0            0            0       
      gi19         71080603      35334        389665    66034313291  
      gi20         3135295       32947        55573      860369696   
      gi21            0            0            0            0       
      gi22          519857        2800         1349      744143539   
      gi23            0            0            0            0       
      gi24            0            0            0            0       
      gi25            0            0            0            0       
      gi26            0            0            0            0       
      gi27            0            0            0            0       
      gi28            0            0            0            0       
      gi29            0            0            0            0       
      gi30            0            0            0            0       
      gi31            0            0            0            0       
      gi32            0            0            0            0       
      gi33            0            0            0            0       
      gi34            0            0            0            0       
      gi35            0            0            0            0       
      gi36         45925184       2868         406      63378633802  
      gi37         82150873        0           5379     76673981442  
      gi38            0            0            0            0       
      gi39          52412          0           1725       6339752    
      gi40            0            0            0            0       
      gi41         4009244       16945        25004      1357622340  
      gi42         9509038       14547         5380     10126148346  
      gi43         66703897      36470        140185    84110324394  
      gi44         5774169       23414        37268      1330076092  
      gi45            0            0            0            0       
      gi46          624994        2800          0        555560603   
      gi47           3652         7920        38886       27129990   
      gi48            0            0            0            0       
      gi49            0            0            0            0       
      gi50         4580230         2          21902      2088276085  
      gi51         60211968     1015841       142797    47584152743  
      gi52        827999694      553863      1346418    869241475537 

      Port       OutUcastPkts OutMcastPkts OutBcastPkts  OutOctets   
---------------- ------------ ------------ ------------ ------------ 
      gi1             0            0            0            0       
      gi2          2683419      2136805      2233347     1323612516  
      gi3             0            0            0            0       
      gi4             0            0            0            0       
      gi5             0            0            0            0       
      gi6           67958       2144137      2242968     493980368   
      gi7             0            0            0            0       
      gi8             0            0            0            0       
      gi9             0            0            0            0       
      gi10            0            0            0            0       
      gi11            0            0            0            0       
      gi12          18003        142172       71711       34736832   
      gi13          104430      2152317      2241351     488183236   
      gi14          115432      2152253      2243063     595853865   
      gi15            0            0            0            0       
      gi16          82882       2151796      2237432     490692620   
      gi17            0            0            0            0       
      gi18            0            0            0            0       
      gi19         77574126     2117062      1853707    62459933755  
      gi20         3651935       645275       656363     3826217283  
      gi21            0            0            0            0       
      gi22          581030       138898       70181      575850550   
      gi23           6239       2152408      2243367     476495882   
      gi24            0            0            0            0       
      gi25            0            0            0            0       
      gi26            0            0            0            0       
      gi27            0            0            0            0       
      gi28            0            0            0            0       
      gi29            0            0            0            0       
      gi30            0            0            0            0       
      gi31            0            0            0            0       
      gi32            0            0            0            0       
      gi33            0            0            0            0       
      gi34            0            0            0            0       
      gi35            0            0            0            0       
      gi36         87095916      74235        224618    65179179062  
      gi37         80372782     2152385      2237959    54348015960  
      gi38            0            0            0            0       
      gi39          50905       2152390      2241628     499351295   
      gi40            0            0            0            0       
      gi41         3349676      2135445      2218347     4004149936  
      gi42         9641295      2137840      2237957     5607606057  
      gi43         90037045     1650927      1490345    85280413173  
      gi44         6102311       550054       560276     7205177135  
      gi45            0            0            0            0       
      gi46          313333       71686        217573     104015424   
      gi47          11636       2144482      2204481     450225824   
      gi48            0            0            0            0       
      gi49            0            0            0            0       
      gi50         3850133      2152399      2221466     1380581066  
      gi51         49740905      638341       383619    34660830247  
      gi52        1299456263     210932       366171    148381739835 
                                                        0            
                                                      
       Ch        InUcastPkts  InMcastPkts  InBcastPkts    InOctets   
---------------- ------------ ------------ ------------ ------------ 
      Po1         888212128     1569734      1489240    916825706399 
      Po2          1144851        5600         1349      1299704142  
      Po3         232927124       5739         430      313085689559 
      Po4             0            0            0            0       
      Po5             0            0            0            0       
      Po6             0            0            0            0       
      Po7             0            0            0            0       
      Po8             0            0            0            0       

       Ch        OutUcastPkts OutMcastPkts OutBcastPkts  OutOctets   
---------------- ------------ ------------ ------------ ------------ 
      Po1         1349197290     849273       749790    151847835892 
                                                        1            
      Po2           894363       210590       287757     679866845   
      Po3          87113919      216411       296333    65213916573  
      Po4             0            0            0            0       
      Po5             0            0            0            0       
      Po6             0            0            0            0       
      Po7             0            0            0            0       
      Po8             0            0            0            0       
switch0f5e86#
```

Po3 is the LAGG in question. gi12 and gi36 are connected to Po3.

Currently there aren't any VLANs. The switch operates in Layer 2 mode.

~Doug


----------

