# Why don't my multi-port LACP interfaces pass more traffic than a single port?



## robroy (Jul 14, 2020)

FreeBSD Friends,

I have two 12.1-RELEASE computers, named _star_ and _joeking_.  Both use LACP with multiple Gigabit Ethernet ports.

*Symptom*

Both _star_ and _joeking_ have the same symptom. 

When I run two simultaneous tests with benchmarks/iperf (from two separate computers), the tests report ~450 Mbit/sec each.

I was hoping to see them report ~900 Mbit/sec each (full Gigabit Ethernet).

_*Questions*_

Is it reasonable for me to expect LACP to distribute a mere two flows across separate links?
If a greater number of flows is required before LACP "kicks in," how can I find out how many?
Does anybody know how to tune LACP to evenly distribute a handful of flows?
_*Stuff I tried*_

I tried swapping the iperf client/server roles (making the clients the servers); 'no change.
I tried using iperf's TCP mode; 'no change.
I tried using two different server-side port numbers with iperf; 'no change.
I tried putting `net.link.lagg.default_use_flowid=1` in /etc/sysctl.conf and rebooting; 'no change.
_*Configurations*

star_ has two Broadcoms attached to "X," an unknown Ethernet switch (I could find out if it really matters).

_joeking_ has four Intels attached to a Cisco Catalyst 2960.

_star_'s rc.conf:
	
	



```
ifconfig_bge0="up"
ifconfig_bge1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bge0 laggport bge1"

vlans_lagg0="11 20 21 37 57 58 59 239"
ifconfig_lagg0_11="up"
ifconfig_lagg0_20="up"
ifconfig_lagg0_21="up"
ifconfig_lagg0_37="inet 172.16.37.194 netmask 255.255.255.0"
ifconfig_lagg0_57="inet 172.16.57.101 netmask 255.255.255.0"
ifconfig_lagg0_58="up"
ifconfig_lagg0_59="up"
ifconfig_lagg0_239="inet 172.16.239.72 netmask 255.255.255.0"
```

_star_'s lagg0

```
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 40:8d:5c:67:73:04
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=10<LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,40-8D-5C-67-73-04,00CB,0000,0000),
                 (8000,58-97-1E-F7-14-80,0001,0000,0000)]
        laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,40-8D-5C-67-73-04,00CB,8000,0001),
                 (8000,58-97-1E-F7-14-80,0001,8000,012F)]
        laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,40-8D-5C-67-73-04,00CB,8000,0002),
                 (8000,58-97-1E-F7-14-80,0001,8000,0130)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

_joeking_'s rc.conf:
	
	



```
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_igb2="up"
ifconfig_igb3="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3"

vlans_lagg0="11 20 21 37 57 58 59 239"
ifconfig_lagg0_11="up"
ifconfig_lagg0_20="up"
ifconfig_lagg0_21="up"
ifconfig_lagg0_37="up"
ifconfig_lagg0_57="inet 172.16.57.239 netmask 255.255.255.0"
ifconfig_lagg0_58="up"
ifconfig_lagg0_59="up"
ifconfig_lagg0_239="up"
```

_joeking_'s lagg0

```
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 90:e2:ba:39:81:e0
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=10<LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 4
                flapping: 0
        lag id: [(8000,90-E2-BA-39-81-E0,018B,0000,0000),
                 (8000,F4-7F-35-BF-78-80,0001,0000,0000)]
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-E2-BA-39-81-E0,018B,8000,0001),
                 (8000,F4-7F-35-BF-78-80,0001,8000,0124)]
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-E2-BA-39-81-E0,018B,8000,0002),
                 (8000,F4-7F-35-BF-78-80,0001,8000,0122)]
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-E2-BA-39-81-E0,018B,8000,0003),
                 (8000,F4-7F-35-BF-78-80,0001,8000,0123)]
        laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,90-E2-BA-39-81-E0,018B,8000,0004),
                 (8000,F4-7F-35-BF-78-80,0001,8000,0125)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```


----------



## `Orum (Jul 16, 2020)

My best guess is that the problem here isn't FreeBSD, but the switch.  L4 hashing on the FreeBSD machines should distribute the sent frames over the various ports (at least for TCP), but once it's in the switch, the switch gets to decide which port it sends the traffic out to to the receiving side.  If it only uses L2 and/or L3 hashing, those will always result in the same port being selected to send the frames over, thus only saturating a single port.

To test this theory, you could just connect the machines directly with a crossover (or straight through if they have auto MDI/MDI-X) and see if you get all the ports saturated.  Or if that's too much hassle, just look at the interface counters on the switch while the traffic to them is otherwise mostly idle, and you can see which interfaces they're arriving on and being sent out on.  If I'm right, you should see them spread around on the receiving interfaces (from the switch's POV), but only being sent out a single port.


----------



## robroy (Jul 16, 2020)

Orum, thank you.

_joeking_ and _star_ are remote and the switches there are run by a busy chap, so I'll try fiddling around at home with a little WS-2960G-8TC-L first.  Hopefully whatever secret sauce makes that work will also solve the _joeking_ and _star_ problem.

I'll update this thread with whatever I find.  I appreciate your time Buddy.


----------



## SirDice (Jul 17, 2020)

robroy said:


> I'll try fiddling around at home with a little WS-2960G-8TC-L first.


In case you haven't seen it yet, there's a Cisco configuration example in the handbook: 31.7.1. Configuration Examples


----------



## PacketMan (Jul 17, 2020)

`Orum said:


> If it only uses L2 and/or L3 hashing, those will always result in the same port being selected to send the frames over, thus only saturating a single port.



I see this 99% of the time.  The general expectation is that the the LACP bundle is carrying traffic to/from multiple sources and/or destinations.  So we between two switches there would be anywhere from dozens to thousands of flows that have unique combinations of source/destination mac/ip addresses.  Between a switch and a server the same logic applies, but the hashing would work on the uniqueness of all the clients addresses.

In some environments, like multicast video, LACP bundles would fail big time....one link would 'burn up' but the rest would be idle. New software on routers/switches have more hashing options to make traffic distribution across the links work.


----------



## robroy (Jul 17, 2020)

SirDice and PacketMan, thank you both.

PacketMan, I'm not sure if I'm interpreting your post correctly; are you saying that you wouldn't expect LACP to use multiple links in my environment (two flows from separate clients)?

I also tried with four flows (from four separate clients), and still didn't see total throughput beyond that of a single GigE link.

Meanwhile, I ordered the WS-2960G-8TC-L on eBay for $38, and it'll arrive around July 24th.  'looking forward to learning more then.


----------



## robroy (Jul 18, 2020)

The toy arrived early, though sadly I lack the right serial cable for its console; one's on order.


----------



## robroy (Jul 25, 2020)

A $2.03 serial cable arrived today:






'ready for action!  The two yellows go to one computer, the two blues to another, and the gray's an uplink to my normal LAN (which amounts to two unmanaged GigE switches).






I plugged the console cable in, ran cu(1) on /dev/cuau0, plugged the Cisco's power cord in, and everything came up on the first try.


```
# cu -l /dev/cuau0     
Connected                                                                                      

Using driver version 1 for media type 1       
Base ethernet MAC Address: 9c:4e:20:7c:17:00                                                   
Xmodem file system is available.
The password-recovery mechanism is enabled.
Initializing Flash...                                                                          
mifs[2]: 0 files, 1 directories
mifs[2]: Total bytes     :    3870720     
mifs[2]: Bytes used      :       1024                                                          
mifs[2]: Bytes available :    3869696
mifs[2]: mifs fsck took 1 seconds.        
mifs[3]: 2 files, 1 directories                                                                
mifs[3]: Total bytes     :   27998208
mifs[3]: Bytes used      :   11932160
mifs[3]: Bytes available :   16066048
mifs[3]: mifs fsck took 5 seconds.
...done Initializing Flash.                                                                    
done.                                                                                          
Loading "flash:c2960-lanbasek9-mz.150-2.SE11.bin"...@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:c2960-lanbasek9-mz.150-2.SE11.bin" uncompressed and installed, entry point: 0x3000
executing...                 
                                               
              Restricted Rights Legend
                                                                                               
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
                                               
           cisco Systems, Inc.                                                                 
           170 West Tasman Drive             
           San Jose, California 95134-1706    
                                               
                                               
                                               
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 15.0(2)SE11, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Sat 19-Aug-17 09:34 by prod_rel_teamInitializing flashfs...           
Using driver version 3 for media type 1                                                        
mifs[4]: 0 files, 1 directories                                                                
mifs[4]: Total bytes     : 3870720   
mifs[4]: Bytes used      : 1024      
mifs[4]: Bytes available : 3869696   
mifs[4]: mifs fsck took 1 seconds.
mifs[4]: Initialization complete.

mifs[5]: 2 files, 1 directories
mifs[5]: Total bytes     : 27998208                                                                                                                                                            
mifs[5]: Bytes used      : 11932160                                                                                                                                                            mifs[5]: Bytes available : 16066048  
mifs[5]: mifs fsck took 0 seconds.                                                             
mifs[5]: Initialization complete.                                                                                                                                                              
                                                                                               
...done Initializing flashfs.                                                                  
Checking for Bootloader upgrade..                                                                                                                                                              
Boot Loader upgrade not required (Stage 2)                                                     
                                               
POST: CPU MIC register Tests : Begin         
POST: CPU MIC register Tests : End, Status Passed                                                                                                                                              
                                                                                               
POST: PortASIC Memory Tests : Begin                                                            
POST: PortASIC Memory Tests : End, Status Passed                                       
                                                                                               
POST: CPU MIC interface Loopback Tests : Begin                                                                                                                                                 
POST: CPU MIC interface Loopback Tests : End, Status Passed                                                                                                                                    
                                                                                                                                                                                               
POST: PortASIC RingLoopback Tests : Begin                                                                                                                                                      
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin
POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

Waiting for Port download...Complete


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960G-8TC-L (PowerPC405) processor (revision C0) with 65536K bytes of memory.
Processor board ID FOC1414X0S3
Last reset from power-on
1 Virtual Ethernet interface
8 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

64K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 9C:4E:20:7C:17:00
Motherboard assembly number     : 73-10613-08
Power supply part number        : 341-0208-01
Motherboard serial number       : FOC14135SAF
Power supply serial number      : LIT140902GT
Model revision number           : C0
Motherboard revision number     : B0
Model number                    : WS-C2960G-8TC-L
System serial number            : FOC1414X0S3
Top Assembly Part Number        : 800-28133-02
Top Assembly Revision Number    : B0
Version ID                      : V02
CLEI Code Number                : COM7S00ARB
Hardware Board Revision Number  : 0x01


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
*    1 8     WS-C2960G-8TC-L    15.0(2)SE11           C2960-LANBASEK9-M        




Press RETURN to get started!


*Mar  1 00:00:35.441: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Mar  1 00:00:35.542: %SMI-5-CLIENT:  Smart Install Client feature is enabled. It is recommended to disable the Smart Install feature when it is not actively used. To disable feature execute 
'no vstack' in configuration mode
*Mar  1 00:00:36.733: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Mar  1 00:00:37.748: %DC-4-FILE_OPEN_WARNING: Not able to open flash:/dc_profile_dir/dc_default_profiles.txt 
*Mar  1 00:00:37.748: %DC-6-DEFAULT_INIT_INFO: Default Profiles DB not loaded. 
*Mar  1 00:00:59.508: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C2960 Software (C2960-LANBASEK9-M), Version 15.0(2)SE11, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Sat 19-Aug-17 09:34 by prod_rel_team
*Mar  1 00:01:01.463: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/7, changed state to up
*Mar  1 00:01:03.031: %LINK-3-UPDOWN: Interface GigabitEthernet0/7, changed state to up
*Mar  1 00:01:03.594: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Mar  1 00:01:04.265: %LINK-3-UPDOWN: Interface GigabitEthernet0/5, changed state to up
*Mar  1 00:01:04.265: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to up
*Mar  1 00:01:04.600: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
*Mar  1 00:01:05.271: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/5, changed state to up
*Mar  1 00:01:05.271: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to up
*Mar  1 00:01:31.058: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
```


----------



## robroy (Jul 28, 2020)

I achieved two simultaneous 940 Mbit/sec flows by retaining all of FreeBSD's defaults, and changing the switch's load balancing option from the default "src-mac," to "src-dst-ip."  Each flow had different source and destination IPs (four IPs in total).

I was not able to achieve balanced distribution with flows that shared a common source or destination IP.  As Orum mentioned, I'm guessing that L4 (TCP/UDP port numbers) would have to be factored in to achieve this, and the WS-2960G-8TC-L cannot do that (though some Cisco products can).

But this is good enough for me (hosting mixed jails, VMs and NFS).

Here's the minimum IOS configuration it took to achieve this:


```
Switch>enable
Switch#show etherchannel load-balance                ! displays the current profile
Switch#configure terminal
Switch(config)#port-channel load-balance ?           ! displays the supported profiles
Switch(config)#port-channel load-balance src-dst-ip  ! chooses "src-dst-ip" instead of "src-mac"
Switch(config)#interface range GigabitEthernet0/1-2  ! configures ports 1 and 2
Switch(config-if-range)#channel-group 1 mode active
Switch(config-if-range)#exit
Switch(config)#interface range GigabitEthernet0/5-6  ! configures ports 5 and 6
Switch(config-if-range)#channel-group 2 mode active
Switch(config-if-range)#exit
```

This shows the running-config after applying the commands above to a factory-fresh WS-2960G-8TC-L.


```
Switch#show running-config
Building configuration...

Current configuration : 1004 bytes
!
! Last configuration change at 00:13:01 UTC Mon Mar 1 1993
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Switch
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
system mtu routing 1500
!
!
!
!
!
!
!
!
!
!
port-channel load-balance src-dst-ip
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
interface Port-channel1
!
interface Port-channel2
!
interface GigabitEthernet0/1
 channel-group 1 mode active
!
interface GigabitEthernet0/2
 channel-group 1 mode active
!
interface GigabitEthernet0/3
!
interface GigabitEthernet0/4
!
interface GigabitEthernet0/5
 channel-group 2 mode active
!
interface GigabitEthernet0/6
 channel-group 2 mode active
!
interface GigabitEthernet0/7
!
interface GigabitEthernet0/8
!
interface Vlan1
 no ip address
 shutdown
!
ip http server
ip http secure-server
!
vstack
!
line con 0
line vty 5 15
!
end
```


----------



## robroy (Aug 9, 2020)

Update:  having solved the symptom at home by switching my WS-2960G-8TC-L from "src-mac" to "src-dst-ip," I tried the same change with _joeking_ (one of the two computers I first posted about).

With "src-dst-ip" on its 2960 I saw iperf traffic being distributed evenly among _joeking_'s four interfaces, so long as I tested with mixed source/destination IP flows.

Later _joeking_ was plugged in to a Catalyst 4500 instead of its original 2960, and I tried the iperf tests again with the "src-dst-port" L4 profile.  On the 4500 with "src-dst-port," I saw fairly even LACP distribution even with the worst case scenario traffic.  This profile's supposed to distribute among LACP ports based on detected TCP/UDP port differences in the flows--yet I saw it distribute traffic even when I forced matching TCP src/dst port numbers among a handful of flows.

I failed to note down the Catalyst 4500's default "port-channel load-balance" profile before choosing "src-dst-port."  But later while eyeballing Cisco documentation I noticed that unlike the 2960, the default's not "src-mac."  I think the 4500's default is "src-dst-ip."  So I probably would have seen good port distribution even without making the change.

I understand that this is all Cisco stuff; the symptom wasn't caused by FreeBSD at all.  Yet I feel like a small addition to Cisco example in the Handbook's LACP page could be helpful--something like, "after configuring LACP, if flows aren't being distributed among multiple ports in the expected way, it may be necessary to configure the Cisco product with _port-channel load-balance_ to distribute traffic based on detected IP addresses and/or TCP/UDP port numbers."


----------



## Phishfry (Oct 8, 2021)

The newer Cisco switches need similar setting for LAGG.
This malformed doc shows it. This also worked on my newer SG500X-48.





						Link Aggregation Group (LAG) Management and Settings on Sx500 Series Stackable Switches
					

This document shows you how to configure the load balancing algorithm, LAG management, and LAG settings on a switch.



					www.cisco.com
				




So you must switch the load balancing algorithm from MAC Address (default) to IP/MAC Address.

I am testing LAGG with 4 port Chelsio T540 to Switch 10G interfaces XG1 XG2 XG3 XG4


----------

