# Slow network performance compared to linux (again)



## grzegorz-derebecki (Jul 29, 2019)

Hi. 

About one Year ago I posted topic about network performance problems in freebsd compared to linux on hetzner.
I wasn't able to solve problem and I forgot about it. Lately things start getting worst. 

I decided to make simple test: 

1. Created to virtual machines on digitalocean. (don't want blame only hetzner for it). One Freebsd 12.0 on Ubuntu 18.04
2. I use the same server location to have similar results
3. Just install iptraf3 on both machines
4. Run test from my home network and from other networks

Results was like expected from my network download speed is very low from Freebsd server compared to Ubuntu. 
This happens only for my home network. It is possible that my provider use some kind of QoS to limit traffic? 
This don't depend on IP address, Maybe some other flags in TCP protocol? 
Maybe there is some kind settings in freebsd that can mimic packets like it was from linux (to verify problem). 


- for udp speed are the same (but very low)
- if i use parallel mode for iperf3 I can get similar results for freebsd and linux
- I tested lots of performance optimalization Freebsd network but none of them help (changing net.inet.tcp.cc.algorithm etc). 

Any ideas? 

*FREEBSD SERVER*

```
~% iperf3 -t 5 -Rc 167.71.134.232
Connecting to host 167.71.134.232, port 5201
Reverse mode, remote host 167.71.134.232 is sending
[  5] local 192.168.1.100 port 54503 connected to 167.71.134.232 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   191 KBytes  1.56 Mbits/sec
[  5]   1.00-2.00   sec   280 KBytes  2.29 Mbits/sec
[  5]   2.00-3.00   sec   144 KBytes  1.18 Mbits/sec
[  5]   3.00-4.00   sec   184 KBytes  1.51 Mbits/sec
[  5]   4.00-5.00   sec   402 KBytes  3.29 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.05   sec  1.25 MBytes  2.08 Mbits/sec   51             sender
[  5]   0.00-5.00   sec  1.17 MBytes  1.97 Mbits/sec                  receiver
```


*UBUNTU SERVER*

```
~% iperf3 -t 5 -Rc 167.71.133.221
Connecting to host 167.71.133.221, port 5201
Reverse mode, remote host 167.71.133.221 is sending
[  5] local 192.168.1.100 port 54516 connected to 167.71.133.221 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.76 MBytes  14.8 Mbits/sec
[  5]   1.00-2.00   sec  2.10 MBytes  17.6 Mbits/sec
[  5]   2.00-3.00   sec  2.40 MBytes  20.1 Mbits/sec
[  5]   3.00-4.00   sec  2.37 MBytes  19.8 Mbits/sec
[  5]   4.00-5.00   sec  2.28 MBytes  19.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  12.0 MBytes  20.2 Mbits/sec   25             sender
[  5]   0.00-5.00   sec  10.9 MBytes  18.3 Mbits/sec                  receiver
```


----------



## obsigna (Jul 29, 2019)

Did you try this?
https://forums.freebsd.org/threads/slow-network-performance-compared-to-linux.67200/#post-398453


----------



## grzegorz-derebecki (Jul 29, 2019)

obsigna said:


> Did you try this?
> https://forums.freebsd.org/threads/slow-network-performance-compared-to-linux.67200/#post-398453



Just tested it. No efects. Still getting less that 2mbps. (checked before set ifconfig_DEFAULT and after). Compared to linux 20mbps


----------



## ommarmol (Apr 6, 2020)

I'm having exactly the same problem. I just loaded a linux image via USB stick and the iperf3 has run at full speed, but returning to FreeBSD it works 10 times slower than in Linux. I tested all recomendations in several posts on this forum, but nothing worked.

Any ideas? How can it be possible having FreeBSD a superior network stack than Linux? (in theory).

Thanks in advance.


----------



## SirDice (Apr 6, 2020)

ommarmol said:


> Any ideas?


Lots of them. I suggest you tell us what you are running, which network cards are involved and how they are configured.


----------



## ommarmol (Apr 6, 2020)

Thank you SirDice for being so inclined to help others!

I'm running a 12.0-RELEASE kernel:
`uname -a`

```
FreeBSD mymachine 12.0-RELEASE-p4 FreeBSD 12.0-RELEASE-p4 GENERIC  amd64
```


Card model:
`dmesg | grep re0`

```
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xee00-0xeeff mem 0xfdefe000-0xfdefe0ff irq 16 at device 4.0 on pci2
re0: Chip rev. 0x10000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 64:70:02:04:a6:a6
re0: netmap queues/slots: TX 1/256, RX 1/256
```


Ifconfig shows:
`ifconfig`

```
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:18:f3:df:3a:91
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 64:70:02:04:a6:a6
        inet 192.168.1.135 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:28:48:f7:00
        inet6 fe80::2bd:28ff:fe48:f700%tap0 prefixlen 64 scopeid 0x4
        inet 100.80.0.1 netmask 0xffffff00 broadcast 100.80.0.255
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 50162
```


The system has one wireless card (not configured) and one vpn configured with OpenVPN.

The route command shows:
`route show default`

```
route to: default
destination: default
       mask: default
    gateway: 192.168.1.1
        fib: 0
  interface: re0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
```

And netstat:
`netstat -r`

```
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS         re0
100.80.0.0/24      link#4             U          tap0
100.80.0.1         link#4             UHS         lo0
localhost          link#3             UH          lo0
192.168.1.0/24     link#2             U           re0
192.168.1.135      link#2             UHS         lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              localhost          UGRS        lo0
localhost          link#3             UH          lo0
::ffff:0.0.0.0/96  localhost          UGRS        lo0
fe80::/10          localhost          UGRS        lo0
fe80::%lo0/64      link#3             U           lo0
fe80::1%lo0        link#3             UHS         lo0
fe80::%tap0/64     link#4             U          tap0
fe80::2bd:28ff:fe4 link#4             UHS         lo0
ff02::/16          localhost          UGRS        lo0
```


There are any other details you would like to consider?

Thanks a lot in advance.


----------



## SirDice (Apr 6, 2020)

ommarmol said:


> I'm running a 12.0-RELEASE kernel


Upgrade to 12.1 first, 12.0 is now end-of-life.

I would suggest using the Intel (em(4)) interface instead of re(4). My personal opinion of Realtek is rather bad.


----------



## ommarmol (Apr 6, 2020)

Ok thank you. Let me check it please.


----------



## ommarmol (Apr 7, 2020)

Well, I'm upgraded to 12.1-RELEASE and the problems remain the same.

I'm using a Realtek ethernet pci card so I can only use the re0 interface as em0 interface is a wireless one, not configured yet.

The most surprising for me is that iperf3 between local machines runs at full 1000mbit/sec speed (regardless of the direction: FreeBSD to Linux or Linux to FreeBSD), but when downloading from Internet, the Linux machine can go at maximum speed while FreeBSD only at one tenth of such maximum.

Any further ideas?

Thanks in advance.


----------



## SirDice (Apr 7, 2020)

ommarmol said:


> I'm using a Realtek ethernet pci card so I can only use the re0 interface as em0 interface is a wireless one


The em(4) interface is an Intel Pro/1000 Gigabit ethernet adapter, that's a _wired_ interface.


----------

