# Poor network performance on FreeBSD 12.1



## Nimce (Dec 30, 2019)

Hi,
I'm running FreeBSD as a guest on the latest VMWare Workstation (15.5.1), the network interface is bridged.
I have installed open-vm-tools-nox11-10.3.0_1,2 (I use console only, no GUI), and for some reason, the network performance, when tested with speedtest cli and iperf3, are almost 50% lower than usual.
I have a gigabit ethernet connection between the VMWare Workstation server and my computer.

Here are the results from iperf3 with FreeBSD 12.1:





And here are the results from a non FreeBSD server, running on the same server, same VMWare Workstation, just different virtual machine:




As you can see, the non FreeBSD server runs almost at 1Gbps speed, and the BSD machine runs run 470-500 Mbps, I have run a few tests on both machine, same results.

Here is the output of `ifconfig` on the FreeBSD 12.1 machine:

```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 00:0c:29:4b:66:b0
        inet 192.168.0.10 netmask 0xffff0000 broadcast 192.168.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

What is the cause of this issue?
Thanks!


----------



## pajaro (Dec 31, 2019)

Hi,

Did you tried: https://calomel.org/freebsd_network_tuning.html ?

I'm experiencing some problems since I updated to 12.1 (VM too): daily hangs, and I need to reboot. Are you experiencing network hangs?

Regards


----------



## Nimce (Dec 31, 2019)

Yes, I have tried, didn't help 
Network hangs I don't encounter, which VM are you using? maybe it's just a problem with VMWare Workstation.


----------



## SirDice (Dec 31, 2019)

Try turning off TSO and/or LRO on the if_em(4) interface.


```
tso     If the driver supports tcp(4) segmentation offloading, enable TSO
             on the interface.  Some drivers may not be able to support TSO
             for ip(4) and ip6(4) packets, so they may enable only one of
             them.

     -tso    If the driver supports tcp(4) segmentation offloading, disable
             TSO on the interface.  It will always disable TSO for ip(4) and
             ip6(4).

     tso6, tso4
             If the driver supports tcp(4) segmentation offloading for ip6(4)
             or ip(4) use one of these to selectively enabled it only for one
             protocol family.

     -tso6, -tso4
             If the driver supports tcp(4) segmentation offloading for ip6(4)
             or ip(4) use one of these to selectively disable it only for one
             protocol family.

     lro     If the driver supports tcp(4) large receive offloading, enable
             LRO on the interface.

     -lro    If the driver supports tcp(4) large receive offloading, disable
             LRO on the interface.
```

Because it's actually a virtual interface TSO/LRO often interferes, especially in combination with PF.


----------



## Nimce (Dec 31, 2019)

Hi,
Thank you for your answer, didn't help, same speeds...

I ran:
`# ifconfig em0 -tso6 -tso -lro`

And then restarted networking.


----------



## obsigna (Dec 31, 2019)

May it be that some sort of local caching is involved on the non-FreeBSD system.

We see for 10 seconds exactly the same maximum rate and the local/sender CPU utilization is much higher than in the FreeBSD case. In contrary, the remote/sender CPU utilization of the non-FreeBSD system is negligibly 0.5 %, which makes me doubt that any significant amounts of data were actually transferred, while the FreeBSD system bumps the CPU to 60 %.


----------



## Eric A. Borisch (Dec 31, 2019)

Nimce said:


> Yes, I have tried, didn't help
> Network hangs I don't encounter, which VM are you using? maybe it's just a problem with VMWare Workstation.



There’s a lot of options they discuss on that page. You tried everything? Including congestion control choices?


----------



## Nimce (Dec 31, 2019)

Yes, did not help either


----------



## Nimce (Dec 31, 2019)

Is there a way to use the vmx network adapter instead of the em adapter? maybe it will perform better?


----------



## gnoma (Jan 2, 2020)

Have you tried different version of FreeBSD?


----------



## Nimce (Jan 2, 2020)

Yes, with FreeBSD 11.3 I experience the exact same speeds.


----------



## gnoma (Jan 2, 2020)

Would it be possible that I/O bottleneck is in happening?
If you are transferring it would be written on a filesystem with some disk device as backend.
Linux OS may apply block device buffering into the RAM and FreeBSD with it's raw devices may be slower on small writes due it's raw device nature.

If you are testing with a file may be a good idea to have try `nc`-ing into /dev/null and `telnet`-ing from /dev/zero or some other scheme that will not involve spinning the disk.


----------



## Nimce (Jan 2, 2020)

I'm testing the speeds with 2 different tools:
`speedtest` (speedtest.net cli)
`iperf3`

I have also tested it with softether vpn server, and I experience lower network speeds with FreeBSD there as well.
I don't think this is an I/O bottleneck, but probably a driver issue, is it possible to force FreeBSD to use another network driver (for example, vmx) when running in a virtual machine?


----------



## Eric A. Borisch (Jan 3, 2020)

Nimce said:


> I'm testing the speeds with 2 different tools:
> `speedtest` (speedtest.net cli)
> `iperf3`
> 
> ...



You need to change the device presented to the VM (from VMware; outside of FreeBSD) from Intel pro/1000 to VMXNET3 to use the vmx(4) driver. Note your network config will need updating for the new device name.


----------



## Nimce (Jan 3, 2020)

Eric A. Borisch said:


> You need to change the device presented to the VM (from VMware; outside of FreeBSD) from Intel pro/1000 to VMXNET3 to use the vmx(4) driver. Note your network config will need updating for the new device name.



Thanks!
Changing to vmx0 did make an improvement, from 400-500 Mbps to 600-700 Mbps, with TSO and LRO disabled as well,
But it's still not what I get from any other operating system running on the same VMWare Workstation (930 Mbps is the expected speed...).

Any ideas?
Thanks


----------



## Eric A. Borisch (Jan 3, 2020)

Output of `ifconfig vmx0` ?

Using _vmx_ is a very different path (from your user-space down to the hardware) than using a virtualized 'device' like _em_; you might try having TSO and LRO on or other network optimizations again. You can also try enabling MSI-X for multiple queues -- be aware that, at least according to the post, you'll need to configure your number of cores (queues) in loader.conf, too. See also vmx(8).

I'd look at the ifconfig or ethtool information on other VM that are performing better, as well.


----------



## Nimce (Jan 3, 2020)

Eric A. Borisch said:


> Output of `ifconfig vmx0` ?
> 
> Using _vmx_ is a very different path (from your user-space down to the hardware) than using a virtualized 'device' like _em_; you might try having TSO and LRO on or other network optimizations again. You can also try enabling MSI-X for multiple queues -- be aware that, at least according to the post, you'll need to configure your number of cores (queues) in loader.conf, too. See also vmx(8).
> 
> I'd look at the ifconfig or ethtool information on other VM that are performing better, as well.



Having TSO and LRO on greatly reduced the performance (500 Mbps with TSO and LRO on, without them, 700 Mbps).
Here is the output of `ifconfig vmx0`:


```
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:4b:66:b0
        inet 192.168.0.10 netmask 0xffff0000 broadcast 192.168.255.255
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

Here is the output of a Linux machine running on a virtual machine (On VMWare Workstation), running on the same server but performing at 930 Mbps:

```
ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.13  netmask 255.255.0.0  broadcast 192.168.255.255
        inet6 fe80::20c:29ff:feb9:af98  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:b9:af:98  txqueuelen 1000  (Ethernet)
        RX packets 767178  bytes 58350480 (55.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 391675  bytes 27898855 (26.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  base 0x2000
```


----------

