# Slow network performance on WAN



## tOsYZYny (Aug 27, 2020)

I have multiple laptops running FreeBSD 12.1 (used as a router, only 1 is active at a time) and all are maxing out at < 100 Mbps and I should be getting 200 Mbps.  I tried the same hardware booted up to Ubuntu and it is seeing > 200 Mbps, so I am sure it isn't hardware.  I also have tried disabling lro, tso, tx and rx checksums, but that didn't seem to have any effect.

I should note that I since I'm using a laptop as a router, I am using a USB Ethernet adapter to provide the WAN connection.  As the laptop is older, it is only USB 2.0 (480 Mbps).  That should still be providing adequate bandwidth to get 200 Mbps.  The LAN port is 1 Gbps.

I ran tcpdump to see what was going on and I see a high number of TCP retransmissions and duplicate acks.  Additionally, when the laptop is idle, the CPU is 99+% idle, whenever I run a speed test, I see the CPU idling at as little as 35%.  I am running ZFS on all the laptops.  Is there perhaps some sort of interaction with ZFS, network load, and CPU usage that is causing this degradation?  If the CPU is having trouble keeping up, that could explain the retransmissions or duplicate acks, it's too latent.  I also tried disabling pf to see if my rules were too complex and that made no difference.


----------



## Mjölnir (Aug 27, 2020)

So you're not connecting via WLAN/WiFi, correct?  You do not have a switch or hub, but instead the USB ethernet adapter connects to the cable modem with an ethernet cable?


----------



## tOsYZYny (Aug 27, 2020)

Yes, this is a wired connection - I'm connecting directly from the Fiber Optic Terminal's Ethernet port to my laptop(s).  I swapped cables, adapters, for whatever reason, FreeBSD is giving me < 100 Mbps whereas Ubuntu was giving me 230 Mbps and I should max out at 200 Mbps.


----------



## Mjölnir (Aug 27, 2020)

Maybe the driver negotiated simplex instead of duplex connection?  What does `ifconfig` show?  It could also be a problem on the USB side.  We need more infos.


----------



## tOsYZYny (Aug 27, 2020)

Ah, good point, it negotiated 100baseTX:

wan: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80008<VLAN_MTU,LINKSTATE>
    ether <OMITTED>
    inet <OMITTED> netmask 0xffffff00 broadcast <OMITTED>
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Walter


----------



## olli@ (Aug 27, 2020)

tOsYZYny said:


> Ah, good point, it negotiated 100baseTX


Well, the maximum for 100baseTX is 100 Mbps.

By the way, you forgot to mention the most important piece of information: What ethernet hardware is that exactly, and which driver is attaching to it?


----------



## mark_j (Aug 27, 2020)

Small point: You'll never get the theoretical maximum speed of a network protocol because of the overhead.


----------



## tOsYZYny (Aug 27, 2020)

Ok, with the adapter that is Gigabit:

wan: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether <OMITTED>
    inet6 <OMITTED>%wan prefixlen 64 scopeid 0x3
    inet <OMITTED> netmask 0xffffff00 broadcast <OMITTED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


It is reporting it negotiated Gigabit; however, when I run fast-cli, I am only seeing about 60 Mbps.  Additionally, the internal NIC is also Gigabit and negotiated as that.  The driver for this particular device is if_ure and uether.

Yes, I understand about overhead and with a 100 Mbit connection, I would be reasonably satisfied for 80 Mbps.  My connection should be 200 Mbps and I am connected at 1 Gbps, so I would expect to see 200 Mbps like I saw in Ubuntu.

The device I am using is a TP-Link 10/100/1000 USB 3.0 adapter.  My laptop only supports USB 2.0, so 480 Mbps:
ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)

Still, I'm only measuring 60 Mbps.


----------



## olli@ (Aug 27, 2020)

I guess that’s the USB overhead, then.
FreeBSD’s USB stack is rather slow, unfortunately.


----------



## Phishfry (Aug 27, 2020)

tOsYZYny said:


> I am using a USB Ethernet adapter to provide the WAN connection.


I bet this is the problem. USB ethernet is not so great.
Have you tried reversing the arrangment and use onboard ethernet as WAN and USB ethernet as LAN?
That would pinpoint the problem.


----------



## mark_j (Aug 28, 2020)

I am not sure of what fast-cli does, but can you perhaps use ftp or some other well known protocol to measure speed? Maybe this fast-cli is just bad on FreeBSD?
Are there other USB devices active on that bus?
ZFS is slower to write than UFS so that will always hamper comparisons.
Is this machine of yours hyperthreaded?
Personally, I would enable the soreceive interface to see if that speeds up your connection:
net.inet.tcp.soreceive_stream=1
in loader.conf
You should see some speed increase. How much?


----------



## tOsYZYny (Aug 28, 2020)

It was the USB Ethernet adapter - I thought I had tried the internal Ethernet port earlier, once I tried it again, I was pulling over 300 Mbps.  So, I guess no more laptops for use as a router, I will use an old desktop since I have multiple Gigabit NICs I can use and they won't have the USB performance hit.

That is surprising, how can the USB stack be that slow?


----------



## mark_j (Aug 28, 2020)

I don't believe the USB stack is, but the Ethernet to USB is, obviously. To clarify, FreeBSD might not fully support your USB dongle. Just a guess.


----------



## tOsYZYny (Aug 28, 2020)

Ok, how might I go about making it better?  Would it be simple enough to make the code changes?  Also, I tried the net.inet.tcp.soreceive_stream=1 in loader.conf and didn't see any marked difference.  I also grepped for stream and tried setting hw.usb.xhci.streams=1.  That too did nothing.


----------



## mark_j (Aug 28, 2020)

What's the name of the dongle/adapter? Do you know the chipset it's using?


----------



## tOsYZYny (Aug 28, 2020)

It is a TP-Link USB 3.0 Gigabit Ethernet Adapter - ure, Realtek RTL8153 .  When running fast-cli, the CPU usage increases from roughly 1% to at least 30%.

see: https://www.mail-archive.com/freebsd-current@freebsd.org/msg181165.html - it seems to indicate that the USB network stack is the problem.  I'm not sure if that can be configured away.  Furthermore, if I am reading it correctly, NetBSD seems to perform better here than FreeBSD.


----------



## mark_j (Aug 28, 2020)

A quick look shows it's supported fine;  well as fine as any realtek chipset can be by FreeBSD.
What's the output of `usbconfig dump_device_desc` ?
The name of the device?


----------



## a6h (Aug 28, 2020)

add media 1000baseTX mediaopt full-duplex to your ifconfig(8) something like this:
`ifconfig interface ip media 1000baseTX mediaopt full-duplex`

[EDIT] from ure(4): RTL8153/1000baseTX: Set 1000baseTX operation over twisted pair. The RealTek gigE chips support 1000Mbps in *full-duplex mode only*


----------



## tOsYZYny (Aug 28, 2020)

ugen1.5: <TP-LINK USB 10/100/1000 LAN> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)

  bLength = 0x0012 
  bDescriptorType = 0x0001 
  bcdUSB = 0x0210 
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000 
  bDeviceProtocol = 0x0000 
  bMaxPacketSize0 = 0x0040 
  idVendor = 0x2357 
  idProduct = 0x0601 
  bcdDevice = 0x3000 
  iManufacturer = 0x0001  <TP-LINK>
  iProduct = 0x0002  <USB 10/100/1000 LAN>
  iSerialNumber = 0x0006  <000001000000>
  bNumConfigurations = 0x0002


----------



## tOsYZYny (Aug 28, 2020)

vigole - thanks, I tried that, but I didn't notice any change.


----------



## ekvz (Aug 28, 2020)

I don't have particularly good experiences with USB3.0 ethernet adapters on USB2.0 ports in general. One of those used to be part of a Linux based router setup i ran a couple of years ago and it gave me nothing but pain (bad connection, randomly disappearing from the bus, ...) until i replaced it with a USB 2.0 one. I have no idea what the exact models were though.

It's not really a direct solution but maybe you could replace the router laptop with an old thinclient (Fujitsu Siemens S400, Igel H700C or something like that). Both of them have gigabit ethernet ports and a usable PCI slot (for more ports) built in. You should be able to get those at around 10€ including shipping on a certain huge auction site.


----------



## mark_j (Aug 28, 2020)

tOsYZYny said:


> ugen1.5: <TP-LINK USB 10/100/1000 LAN> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)
> 
> bLength = 0x0012
> bDescriptorType = 0x0001
> ...


Ok, so how is it set up? You have a devd setting for this unit?


----------



## tOsYZYny (Aug 28, 2020)

Uh, I don't think so.  The only thing I do on my systems is rename the interfaces and configure static IP or dhcp.  In this case here, I think I just renamed the interface using ifconfig_ure0_name=wan.  Then, I instructed it to use dhcp.


----------



## mark_j (Aug 28, 2020)

Who/what  instructed you to do that?
What does dmesg report when the device is plugged in?
You will note your device has multiple configurations. Have you tried _ usbconfig -d 1.5 set_config 1 ?_


----------



## tOsYZYny (Aug 28, 2020)

I like to rename my interfaces so they're intuitive to me.

ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6
ure0 on uhub0
ure0: <TP-LINK USB 10/100/1000 LAN, class 0/0, rev 2.10/30.00, addr 2> on usbus6
miibus1: <MII bus> on ure0
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
ue0: <USB Ethernet> on ure0
ue0: Ethernet address: <OMITTED>
ue0: link state changed to DOWN
ue0: changing name to 'wan'
wan: link state changed to UP
wan: link state changed to DOWN
wan: link state changed to UP
wan: link state changed to DOWN
wan: link state changed to UP

Changing the config to 1 caused the device to disappear ...


----------



## mark_j (Aug 28, 2020)

tOsYZYny said:


> I like to rename my interfaces so they're intuitive to me.
> 
> ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6
> ure0 on uhub0
> ...


oh ok, it's just impossible to discern the device driver when it's called 'wlan' that's all


----------



## mark_j (Aug 30, 2020)

tOsYZYny said:


> I like to rename my interfaces so they're intuitive to me.
> 
> ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6
> ure0 on uhub0
> ...


What was the log message?


----------



## tOsYZYny (Aug 30, 2020)

I ran that command again to try to pull logs and it didn't make the device disappear, but it did cause it to lose its IP.  I was able to renew it and functionality worked as normal.  However, there was no difference in bandwidth after running that command.

I can see that usbconfig dump_all_config_desc shows the 2 configurations, but can't make heads or tails of what it means.


----------



## freebsdtvsrv (Oct 21, 2020)

same trouble with same TP-LINK USB-ETHERNET 1Gbps adapter, but in USB3.0 port

test of http download:
# fetch http://10.9.9.5/sized.apk
fetch: http://10.9.9.5/sized.apk: size of remote file is not known
sized.apk                                               76 MB    9 MBps    08s

test via internal ethernet adapter under 50%load :
 # fetch http://10.9.8.5/sized.apk
fetch: http://10.9.8.5/sized.apk: size of remote file is not known
sized.apk                                               76 MB   37 MBps    02s


----------



## olli@ (Oct 22, 2020)

freebsdtvsrv said:


> same trouble with same TP-LINK USB-ETHERNET 1Gbps adapter, but in USB3.0 port
> 
> test of http download:
> # fetch http://10.9.9.5/sized.apk
> ...


That’s the speed I would expect from a USB ethernet adapter with FreeBSD.

If you need more speed, buy a PCIe NIC. If you can’t do that, but you still need more ports, buy a network switch and connect it to the internal port. The 1 Gbps will be shared in this case, but that’s still better than USB ethernet. If you need to control the ports from FreeBSD (e.g. for routing, IPFW rules, whatever), choose a VLAN-capable switch and configure a VLAN trunk on the FreeBSD machine, then you’ll get the switch ports as separate virtual interfaces vlan0, vlan1, vlan2 etc.

By the way, you forgot to show the output from ifconfig(8) and usbconfig(8). Related lines from dmesg(8) could be useful, too.


----------



## tOsYZYny (Oct 26, 2020)

I switched to PCIe and am getting what I expect.  I was hoping to use USB NICs so that I could have an extra battery backup by taking advantage of the laptop battery.


----------

