# Asymmetric Network Performance FreeBSD VM on XCP-NG



## Daryl Bert (Aug 17, 2020)

I have a Dell R720 with XCP-NG 8.1 installed on it -- After installing a FreeBSD 11.3 virtual machine I noticed that I was getting asymmetric network transfer speeds between VM machines. I have several Debian and Ubuntu virtual machines that have symmetric network performance to one another. However, when a network test is run between the FreeBSD virtual machine and the Debian/Ubuntu virtual machines the transfer shows asymmetric results. i.e. one direction is much slower than the other. Furthermore, FreeBSD VM (post version 10.4) to FreeBSD VM (post version 10.4) demonstrates symmetric performance but slow in both directions. After identifying the asymmetry I decided to install older and newer versions of FreeBSD and my research shows that FreeBSD 10.4 actually performs at a higher level than my Linux based VMs and is symmetric (or closer to symmetric - both are acceptable performance for me).

FreeBSD 11.0 is the version where the asymmetry appears and it also exists in 11.3 and 12.1. i.e. it is still the case for the most recent production release.

`dmesg | grep xn0` (FreeBSD VM)

Output:
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: xx:xx:xx:xx:xx:xx
xn0: backend features:random: unblocking device.
xn0: 2 link states coalesced
xn0: link state changed to UP

*Network performance - measured via iperf3

FreeBSD 11.3 (similar results FreeBSD 11.0 and 12.1) (172.16.1.56) <--> Ubuntu 18.04 (172.16.1.4) (Asymmetric performance)*

`iperf3 -c 172.16.1.4`

Output: (poor performance compared to next test going the other direction - notice the large number of retries in the results)
Connecting to host 172.16.1.4, port 5201
[  5] local 172.16.1.56 port 41608 connected to 172.16.1.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   293 MBytes  2.46 Gbits/sec  469    211 KBytes     
[  5]   1.00-2.00   sec   288 MBytes  2.41 Gbits/sec  352    216 KBytes     
[  5]   2.00-3.00   sec   284 MBytes  2.38 Gbits/sec  665    156 KBytes     
[  5]   3.00-4.00   sec   276 MBytes  2.31 Gbits/sec  344   90.7 KBytes     
[  5]   4.00-5.00   sec   308 MBytes  2.59 Gbits/sec  724    154 KBytes     
[  5]   5.00-6.00   sec   304 MBytes  2.55 Gbits/sec  813    200 KBytes     
[  5]   6.00-7.00   sec   311 MBytes  2.61 Gbits/sec  676    109 KBytes     
[  5]   7.00-8.00   sec   312 MBytes  2.61 Gbits/sec  490    149 KBytes     
[  5]   8.00-9.00   sec   301 MBytes  2.52 Gbits/sec  670    175 KBytes     
[  5]   9.00-10.00  sec   299 MBytes  2.51 Gbits/sec  598    190 KBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.90 GBytes  2.50 Gbits/sec  *5801*             sender
[  5]   0.00-10.00  sec  2.90 GBytes  2.49 Gbits/sec                  receiver

iperf Done.


`iperf3 -c 172.16.1.4 -R`

Output (higher performance - no retries):

Connecting to host 172.16.1.4, port 5201
Reverse mode, remote host 172.16.1.4 is sending
[  5] local 172.16.1.56 port 50635 connected to 172.16.1.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.08 GBytes  9.23 Gbits/sec                
[  5]   1.00-2.00   sec  1.05 GBytes  9.04 Gbits/sec                
[  5]   2.00-3.00   sec  1.09 GBytes  9.37 Gbits/sec                
[  5]   3.00-4.00   sec  1.06 GBytes  9.15 Gbits/sec                
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec                
[  5]   5.00-6.00   sec  1.13 GBytes  9.75 Gbits/sec                
[  5]   6.00-7.00   sec  1.12 GBytes  9.58 Gbits/sec                
[  5]   7.00-8.00   sec  1.03 GBytes  8.84 Gbits/sec                
[  5]   8.00-9.00   sec  1.12 GBytes  9.60 Gbits/sec                
[  5]   9.00-10.00  sec  1.01 GBytes  8.65 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.27 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.8 GBytes  9.26 Gbits/sec                  receiver

iperf Done.

*FreeBSD 10.4 (172.16.1.54) <--> Ubuntu 18.04 (172.16.1.4) (Good performance both directions)*

`iperf3 -c 172.16.1.4`

Output: (no retries)
Connecting to host 172.16.1.4, port 5201
[  5] local 172.16.1.54 port 52160 connected to 172.16.1.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.62 GBytes  13.9 Gbits/sec    0   3.00 MBytes     
[  5]   1.00-2.00   sec  1.76 GBytes  15.1 Gbits/sec    0   3.00 MBytes     
[  5]   2.00-3.00   sec  1.61 GBytes  13.9 Gbits/sec    0   3.00 MBytes     
[  5]   3.00-4.00   sec  1.57 GBytes  13.5 Gbits/sec    0   3.00 MBytes     
[  5]   4.00-5.00   sec  1.58 GBytes  13.5 Gbits/sec    0   3.00 MBytes     
[  5]   5.00-6.00   sec  1.67 GBytes  14.4 Gbits/sec    0   3.00 MBytes     
[  5]   6.00-7.00   sec  1.70 GBytes  14.6 Gbits/sec    0   3.00 MBytes     
[  5]   7.00-8.00   sec  1.62 GBytes  13.9 Gbits/sec    0   3.00 MBytes     
[  5]   8.00-9.00   sec  1.67 GBytes  14.3 Gbits/sec    0   3.00 MBytes     
[  5]   9.00-10.00  sec  1.68 GBytes  14.5 Gbits/sec    0   3.00 MBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.5 GBytes  14.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.5 GBytes  14.2 Gbits/sec                  receiver

iperf Done.

`iperf3 -c 172.16.1.4 -R`

Output: (no retries)

Connecting to host 172.16.1.4, port 5201
Reverse mode, remote host 172.16.1.4 is sending
[  5] local 172.16.1.54 port 17523 connected to 172.16.1.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.35 Gbits/sec                
[  5]   1.00-2.00   sec  1.07 GBytes  9.16 Gbits/sec                
[  5]   2.00-3.00   sec  1.01 GBytes  8.64 Gbits/sec                
[  5]   3.00-4.00   sec  1.16 GBytes  9.92 Gbits/sec                
[  5]   4.00-5.00   sec  1.01 GBytes  8.70 Gbits/sec                
[  5]   5.00-6.00   sec  1.04 GBytes  8.93 Gbits/sec                
[  5]   6.00-7.00   sec  1.07 GBytes  9.16 Gbits/sec                
[  5]   7.00-8.00   sec  1.08 GBytes  9.24 Gbits/sec                
[  5]   8.00-9.00   sec  1.01 GBytes  8.70 Gbits/sec                
[  5]   9.00-10.00  sec  1.05 GBytes  9.05 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.6 GBytes  9.09 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.6 GBytes  9.09 Gbits/sec                  receiver

iperf Done.

The implication of the asymmetry is that I am getting poor read performance when I am copying files from the FreeBSD 11.3 virtual machine. Based on research in forums I have attempted the three following things independently of one another.

1) Disabling the following on the FreeBSD network interface (rxcsum, txcsum, tso, and lro) using:

`ifconfig xn0 -rxcsum -txcsum -tso -lro`

This did have some positive impact with the speed moving from around 2.5 Gbits/second (from the first test above) to 4.2 Gbits/second. An improvement, but still well short of the 9 Gbit/sec in the other direction).

2) Disabling these features on the virtual network interface in XCP-NG with similar results. Output below shows the state after making the change in XCP-NG.

`xe vif-param-get uuid=b8363445-bab2-dddd-b1234-815fd305cd29 param-name=other-config`

Output:
ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off

3) I have also attempted some sysctl type adjustments with no enhancement in network performance.


I captured some tcpdump data and the significant difference between "good" performance and "poor" performance is the size of the packet. See data below...

*tcpdump data (good performance - FreeBSD 10.4 VM to Ubuntu 18.04 VM) *-- note packet length of 65160

00:36:25.660150 IP (tos 0x0, ttl 64, id 57165, offset 0, flags [DF], proto TCP (6), length 65212)
    172.16.1.54.49266 > 172.1.16.4.5201: Flags [.], cksum 0x590a (incorrect -> 0x99c1), seq 2382579110:2382644270, ack 1, win 1040, options [nop,nop,TS val 1177932 ecr 1428452870], length *65160

tcpdump data (poor performance - FreeBSD 12.1 VM to Ubuntu 18.04 VM)* -- note packet length of 1448

00:34:23.568384 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
    172.16.1.56.63487 > 172.16.1.4.5201: Flags [.], cksum 0x602b (incorrect -> 0x6791), seq 200409586:200411034, ack 1, win 4, options [nop,nop,TS val 14000446 ecr 3327744086], length *1448*

Any ideas of adjustments that can be made in newer FreeBSD versions (11.3 to 12.1) to achieve better network performance in both directions for my VMs?


----------



## Daryl Bert (Aug 18, 2020)

After many searches and reading I am convinced that the packet size is a significant clue. 

My hypothesis is that the VM based network interface (virtio/vtnet in my case I believe) or perhaps netmap may be contributing to the asymmetric transfer speeds. i.e. something in FreeBSD 10.4 allowed the packet sizes to grow from the MTU length of 1500 to 65160 to maximize transfer speeds. FreeBSD 11.0 to current versions are locking the packet size to the MTU length of 1500.

A few questions that I have are...
- what kernel component supports the growth in packet sizes? is it something negotiated between the FreeBSD host and hypervisor?
- are there any diagnostic tests that I can run to either support or refute my admittedly broad hypothesis?

Are there any other hypotheses to dive into?


----------

