# Bad ping/network performance



## Zer0 (Apr 18, 2012)

Hello,

I just installed FreeBSD 9 amd64 on an Asus P8Z77-M Motherboard with the Intel i3-3200 CPU. For networking I use a Intel Pro/1000GT. I also tried an older USRobotics Realtek card, but same thing there.

The problem is, that there are some problems with the network connection, I have pings around 1000 to 3000ms and therefore downloading or ssh sessions are nearly impossible.


```
PING google.de (173.194.69.94): 56 data bytes
64 bytes from 173.194.69.94: icmp_seq=0 ttl=50 time=3998.035 ms
64 bytes from 173.194.69.94: icmp_seq=1 ttl=50 time=2998.279 ms
64 bytes from 173.194.69.94: icmp_seq=2 ttl=50 time=1997.430 ms
64 bytes from 173.194.69.94: icmp_seq=3 ttl=50 time=996.550 ms
64 bytes from 173.194.69.94: icmp_seq=4 ttl=50 time=3992.210 ms
64 bytes from 173.194.69.94: icmp_seq=5 ttl=50 time=2992.898 ms
64 bytes from 173.194.69.94: icmp_seq=6 ttl=50 time=1992.039 ms
64 bytes from 173.194.69.94: icmp_seq=7 ttl=50 time=991.127 ms
64 bytes from 173.194.69.94: icmp_seq=8 ttl=50 time=3988.262 ms
64 bytes from 173.194.69.94: icmp_seq=9 ttl=50 time=2989.221 ms
64 bytes from 173.194.69.94: icmp_seq=10 ttl=50 time=1988.325 ms

PING 192.168.178.1 (192.168.178.1): 56 data bytes
64 bytes from 192.168.178.1: icmp_seq=0 ttl=64 time=3988.928 ms
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=2990.695 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=64 time=1989.759 ms
64 bytes from 192.168.178.1: icmp_seq=3 ttl=64 time=988.842 ms
64 bytes from 192.168.178.1: icmp_seq=4 ttl=64 time=3985.290 ms
64 bytes from 192.168.178.1: icmp_seq=5 ttl=64 time=2986.033 ms
64 bytes from 192.168.178.1: icmp_seq=6 ttl=64 time=1985.200 ms
64 bytes from 192.168.178.1: icmp_seq=7 ttl=64 time=984.180 ms
64 bytes from 192.168.178.1: icmp_seq=8 ttl=64 time=3981.302 ms
64 bytes from 192.168.178.1: icmp_seq=9 ttl=64 time=2982.069 ms
64 bytes from 192.168.178.1: icmp_seq=10 ttl=64 time=1981.054 ms

localhost has no excessive ping responses:
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.201 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.125 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.113 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.090 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.155 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.125 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.113 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.089 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.117 ms
64 bytes from 127.0.0.1: icmp_seq=10 ttl=64 time=0.091 ms
```


My *ifconfig*:

```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 90:e2:ba:0c:8e:33
        inet 192.168.178.100 netmask 0xffffff00 broadcast 192.168.178.255
        inet6 fe80::92e2:baff:fe0c:8e33%em0 prefixlen 64 scopeid 0x3
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

*netstat* for em0 shows no errors or drops on the network:

```
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
usbus     0 <Link#1>                               0     0     0        0     0     0
usbus     0 <Link#2>                               0     0     0        0     0     0
em0    1500 <Link#3>      90:e2:ba:0c:8e:33     4995     0     0     4363     0     0
em0    1500 192.168.178.0 192.168.178.100       4695     -     -     4175     -     -
em0    1500 fe80::92e2:ba fe80::92e2:baff:f        0     -     -        1     -     -
usbus     0 <Link#4>                               0     0     0        0     0     0
lo0   16384 <Link#5>                               0     0     0        0     0     0
lo0   16384 localhost     ::1                      0     -     -        0     -     -
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -
lo0   16384 your-net      localhost                0     -     -        0     -     -
```

I tested the hardware with another linux operating system and networking runs just fine.

Any help is appreciated 

Regards, Paul


----------



## SirDice (Apr 18, 2012)

Try a different cable and/or switch port. Also verify if there aren't any errors on the switch.


----------



## bbzz (Apr 18, 2012)

To be honest, even your lo0 ping looks high to me. It should be more like 0.020ms, not 0.200ms.


----------



## DutchDaemon (Apr 18, 2012)

Install net/mtr-nox11, run [cmd=]mtr 173.194.69.94[/cmd] and see where the actual degradation takes place. If it's always within the local network, then (indeed) check cables, switch ports.


----------



## Zer0 (Apr 18, 2012)

The first thing I tried, was using a different cable and I plugged the network directly into the router. I will try the mtr tool and get back to you.


----------



## SirDice (Apr 18, 2012)

Looking at the pings you already posted I'm quite sure the issue is with your local network. The pings to 192.168.178.1 (I assume that's your router) are already quite high.


----------



## Zer0 (Apr 18, 2012)

I traced the connection over my router:

First to my router
`# mtr 192.168.178.1`

```
My traceroute  [v0.82]
Arcos (0.0.0.0)                                        Wed Apr 18 15:37:49 2012
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.178.1                     5.9%    18  727.3 2119. 325.2 3746. 1155.
```
The corresponding wireshark tracefile http://www.mediafire.com/?2pucd99641698rw

and then to google
`# mtr 173.194.69.94`

```
My traceroute  [v0.82]
Arcos (0.0.0.0)                                        Wed Apr 18 15:35:31 2012
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.178.1                    71.4%    43  2276. 1903.  22.4 3337. 1123.
    87.186.224.34
87.187.186.224.34                    73.8%    43  2184. 2323. 1048. 3921. 947.6
87.187.190.167.34                    69.0%    43  2092. 2189.  72.2 3911. 1169.
hh-ehh-ea7-i.HH.DE.NET.DTAG.DE   26.073.8%    43  2000. 2646. 796.9 3996. 992.0
80.180.150.171.130                   71.4%    43  1908. 2661. 671.0 3932. 1032.
209.209.85.249.132                   69.0%    43  1816. 2426. 545.0 3840. 1086.
72.172.14.233.216                    69.0%    43  1724. 2328. 419.1 3748. 1088.
209.209.85.240.154                   73.8%    43  1632. 2540. 1300. 3656. 856.0
64.264.233.174.29                    78.9%    39  1540. 2537. 1540. 3565. 848.8
    64.233.174.53
10. ???
bk-ibk-in-f94.1e100.net               2.6%    39  783.3 1867.  45.0 3819. 1123.
```
and the wireshark tracefile http://www.mediafire.com/?z0j437wrqvls8d7

In both files, you will see some traffic from pkg_add, but the main thing I notice, is, that the google pings generate lots of ttl exceeded in transit. Other then that, at least to me nothing suspicious. 

I don't get ttl exceeded messages from the pings to my router, but the ping is way higher then normal, but why? I don't have these problems with my Windows PC and my debian laptop. Also the network connection runs fine if I use debian on the PC in question here. I hope you have an idea here.


----------



## DutchDaemon (Apr 18, 2012)

Try [cmd=]mtr -r -c 60 somehost[/cmd] for better output. It will give you a nice report after 60 seconds, which will look better.


----------



## bbzz (Apr 18, 2012)

Show output of:
`# vmstat -i`
`# sysctl net.inet.ip.intr_queue_drops`


----------



## Zer0 (Apr 18, 2012)

`mtr -r -c 60 192.168.178.1`

```
HOST: Arcos                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.178.1              5.0%    60  671.2 2189. 155.1 3929. 1152.2
```

`mtr -r -c 60 193.99.144.80`

```
HOST: Arcos                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.178.1             81.7%    60  3034. 1768.   6.1 4104. 1287.9
    |  `|-- 87.186.224.34
  2.|-- 87.186.224.34             81.7%    60  2956. 2403. 870.6 4026. 1222.3
  3.|-- 87.190.167.46             81.7%    60  2878. 2311. 726.6 3948. 1224.5
  4.|-- f-ed4-i.F.DE.NET.DTAG.DE  81.7%    60  2800. 2219. 582.6 3870. 1227.2
  5.|-- 217.243.218.38            83.3%    60  2722. 2259. 438.7 3792. 1212.0
  6.|-- heise2.f.de.plusline.net  81.7%    60  2644. 2398. 294.7 4728. 1390.1
  7.|-- ???                       100.0    60    0.0   0.0   0.0   0.0   0.0
```

`# vmstat -i`

```
interrupt                          total       rate
irq16: em0 xhci0+                  21094          1
irq23: ehci1                       31492          2
cpu0:timer                       1769602        124
irq256: hdac0                         48          0
irq257: ahci0                     590988         41
cpu1:timer                        292682         20
cpu3:timer                        365865         25
cpu2:timer                       1029205         72
Total                            4100976        288
```

`# sysctl net.inet.ip.intr_queue_drops`

```
net.inet.ip.intr_queue_drops: 0
```


----------



## aa (Apr 18, 2012)

You've installed AMD64 on new INTEL's Ivy-Bridge board, and yet they still worked? *T*hat's weird.


----------



## Zer0 (Apr 18, 2012)

What*'*s weird about that? AMD64 just means, that I installed the 64bit version of FreeBSD. I can't install ia64, cause that is for Intel Itanium CPUs and why should I install i386 when I want to use 16GB of RAM and don't want to use PAE. Or did I miss something?


----------



## bbzz (Apr 18, 2012)

You are correct, disregard previous comment.

Does your motherboard have integrated LAN port you can test?


----------



## aa (Apr 18, 2012)

Sorry, you right, ia64 is for Itanium.


----------



## Zer0 (Apr 18, 2012)

There is a Realtek onboard, but it doesn't work out of the box (neither with FreeBSD nor Debian) and is currently deactivated. Maybe I can get it working, but I'm not that familiar with FreeBSD as I'm with Debian, so I might need some time. I will try it tomorrow cause I'm at work right now.


----------



## DutchDaemon (Apr 18, 2012)

Also try a number of ping and traffic tests _to_ this machine.


----------



## SirDice (Apr 19, 2012)

Also, if you have any, try to ping another machine on your network, not the router. As it may be the router that's causing the delays.


----------



## Zer0 (Apr 19, 2012)

Here is a mtr ping from the FreeBSD box to my local Windows machine:

```
HOST: Arcos                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.178.21             6.7%    60  796.8 2322. 796.8 3847. 1126.5
```

And here a ping using WinMTR from the Windows PC to the BSD box:

```
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                         192.168.178.100 -    0 |   78 |   78 |    0 | 1690 | 3999 |    0 |
|________________________________________________|______|______|______|______|______|______|
```

Also ssh is nearly unusable, it's like I would connect over a 4800 baud line or worse.

I'm sure it has nothing to do with my local network. The only common denominator for this problem is the FreeBSD installation. Debian and Windows installations running smooth on the same hardware, without any modification to the hardware or the network. Only FreeBSD has this strange network problems. But since I'm not the FreeBSD or network guru, I hope you guys here have an idea on how to tackle this


----------



## Anonymous (Apr 19, 2012)

Only a wild guess, however, it's worth to give it a quick try.

Switch, net.inet.tcp.delayed_ack to 0, since the equivalent out-of-the-box setting on Windows is also 0. If it doesn't help, then set it back to 1.


----------



## Zer0 (Apr 19, 2012)

Sadly no, it didn't help


----------



## SirDice (Apr 19, 2012)

What does WinMTR report when you run it against 192.168.178.1?


----------



## aa (Apr 19, 2012)

It's a new board, slighty different architecture. Just an upgrade from PCI2 to PCI3 bus may raise incompatibility to kernel or device drivers. In fact that internal network card is not working in the first place might be an indication of it.

Come on, FreeBSD is not known as a very welcomer to new hardware.


----------



## Zer0 (Apr 19, 2012)

WinMTR from my Windows Box:

```
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                           192.168.178.1 -    0 |  130 |  130 |    0 |    1 |   18 |    1 |
|________________________________________________|______|______|______|______|______|______|
```


----------



## acheron (Apr 19, 2012)

em0 shares its IRQ with xhci and other(s) device(s), maybe this is the root cause of your problem, try to use MSI or disable USB and see how it goes.


----------



## Zer0 (Apr 19, 2012)

I disabled as much as I could, but USB can't be completely disabled
`vmstat -i`

```
interrupt                          total       rate
irq16: em0 ehci0                    1051          1
irq23: ehci1                        1715          2
cpu0:timer                         51069         73
irq256: mps0                         371          0
irq257: ahci0                       1073          1
cpu1:timer                          2471          3
cpu3:timer                          3022          4
cpu2:timer                          8827         12
Total                              69599        100
```
Is there a way to kind of remap the IRQs?


----------



## acheron (Apr 19, 2012)

Can you show us the relevant line of your dmesg about your em0 device and the result of 
	
	



```
sysctl hw.em.enable_msix
```
You can also try to disable rxcsum/txcsum.


----------



## DutchDaemon (Apr 19, 2012)

You could also try to use polling for em0, but you'd have to compile a new kernel for that.


----------



## Zer0 (Apr 19, 2012)

dmesg for my em0 device:

```
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xd000-0xd03f mem 0xf7c40000-0xf7c5ffff,0xf7c20000-0xf7c3ffff irq 16 at device 0.0 on pci4
em0: Ethernet address: 90:e2:ba:0c:8e:33
```
The whole dmesg output: http://pastie.org/private/yfsqzzkxgfgupj5fsfhzq


```
hw.em.enable_msix: 1
```

Disabling rxcsum/txcsum doesn't have any impact on performance/ping.


----------



## acheron (Apr 19, 2012)

You may find the following thread interesting http://lists.freebsd.org/pipermail/freebsd-net/2011-April/028742.html


----------



## Zer0 (Apr 19, 2012)

I would love to get speeds anywhere near 600mbps, I'll get blazing speeds around 10kBps  Here is some output suggested by the ml:


```
dev.em.0.tx_desc_fail2: 0
```

`# vmstat -z|egrep 'ITEM|mbuf'`

```
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
mbuf_packet:            256,      0,     256,     640,    1486,   0,   0
mbuf:                   256,      0,       1,     513,     932,   0,   0
mbuf_cluster:          2048,  25600,     896,      60,     896,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,      14,       8,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
```

`# pciconf -lvc`

```
em0@pci0:4:0:0:	class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82541PI Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction
```
Complete pciconf output here http://pastie.org/private/0m6t36hizxvkrhm76wi31w

I tried the suggested sysctl options from the ml, but sadly they don't have any effect on the performance...


----------



## bbzz (Apr 20, 2012)

Have you tried *polling* yet? 
Not sure if it would matter on Intel cards but could on Realtek.


----------



## Zer0 (Apr 21, 2012)

No, but I found the source of the problem. It's the mainboard in "interaction" with any network card plugged into the PCI port. Debian for instance also has a problem with the board and an extra network card. If I switch the mainboard, then the problem is gone... I should have known it, that a brand new mainboard wouldn't be the best option for a linux/unix based server...
Many thanks to all of you who helped me out here.

Next step: build my own kernel with network support for the onboard network card.


----------



## bbzz (Apr 21, 2012)

While you are at it, build with polling and try, I'm just curious. 
I had relatively new motherboard with similar problem (any network PCI card would instantaneously jump to 50% CPU utilization, while on-board card worked). Polling solved it.

You just need extra line or two* in kernel:


```
options DEVICE_POLLING
options HZ=1000
```

*I think most hardware is @ 1000 Hz so this wouldn't matter.


----------



## free-and-bsd (Jul 30, 2013)

*[FreeNAS] slightly off topic*

Hey, @Zer0 I know it's a bit outdated now, but I'm looking for supporting information on whether a Z77 motherboard's SATA controller will be recognized by FreeNAS.

So your post implies it will, seeing that FreeNAS is now v9.1. Do you have any updates on how your system is functioning?
Since I couldn't find anything in the FreeBSD hardware support list regarding Z77 support, I have to ask those having tried...

Thanks in advance for your answer )


----------



## SirDice (Jul 30, 2013)

free-and-bsd said:
			
		

> Hey, @Zer0 I know it's a bit outdated now, but I'm looking for supporting information on whether a Z77 motherboard's SATA controller will be recognized by FreeNAS.


I have an Asus Sabertooth Z77. It works just fine with FreeBSD 9.x.


----------



## Zer0 (Jul 30, 2013)

free-and-bsd said:
			
		

> Hey, @Zer0 I know it's a bit outdated now, but I'm looking for supporting information on whether a Z77 motherboard's SATA controller will be recognized by FreeNAS.
> 
> So your post implies it will, seeing that FreeNAS is now v9.1. Do you have any updates on how your system is functioning?
> Since I couldn't find anything in the FreeBSD hardware support list regarding Z77 support, I have to ask those having tried...
> ...



I switched to Debian since I'm way more familiar with it so I can't comment on your question. But it should work, @SirDice answer suggests this too.


----------



## free-and-bsd (Jul 31, 2013)

Ha-ha, that's great, guys, thanks a lot .

And me, I'm really getting dull ... I even have 2 machines in my office just recently built (by my own self!) on the basis of Z77-based mobos, so there's no reason why I couldn't just insert a FreeNAS CD (or a FreeBSD for that matter) into either of them and see how it boots and recognizes the hardware... Hope you will excuse me.

@Zer0, understand you perfectly well, because I've done this move many times myself. Yet, with Linux advancing further and further down the path it's chosen I'm coming to love FreeBSD more and more. Which results in my getting to know it better, too.


----------



## free-and-bsd (Aug 9, 2013)

OK, since I've mentioned this, I can also confirm now that FreeNAS _does_ support Z77 (I'm using the motherboard MSI Z77A-G43) even in its present stable version 8.3.1.


----------

