# Slow Intel 10GbE CX4 adapter behaviour



## rihad (Feb 9, 2011)

(sorry for cross-posting this here and to freebsd-net@ in case they have different audiences)

Hi, we're a medium sized ISP that need to pass all incoming user traffic through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until yesterday it had two 1 gb em cards, one for input, one for output. As we were approaching the bandwidth limitation we switched the cards for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. Also, there were many "IP fragmentation failed" errors (1-30 kpps in "*systat -ip*"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough opportunity to test the newer FreeBSD under load, but from the time we did and I know, the same slowness started happening at about 300-400 mbps load. There are no more fragmentation failed errors. No evident drops as per "*netstat -s | fgrep drop*". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 and ix1 are configured at their default settings.

Then I read something about the number of ixgbe device descriptors (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to 4096 permittable. But after some grepping on the source tree I saw that contrary to what the old docs say they are both set to an optimal value:

/sys/dev/ixgbe/ixgbe.c:

```
/*
** Number of TX descriptors per ring,
** setting higher than RX as this seems
** the better performing choice.
*/
static int ixgbe_txd = PERFORM_TXD;
TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);

/* Number of RX descriptors per ring */
static int ixgbe_rxd = PERFORM_RXD;
TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
```


```
/sys/dev/ixgbe/ixgbe.h:
/*
 * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
 * number of transmit descriptors allocated by the driver. Increasing this
 * value allows the driver to queue more transmits. Each descriptor is 16
 * bytes. Performance tests have show the 2K value to be optimal for top
 * performance.
 */
#define DEFAULT_TXD     1024
#define PERFORM_TXD     2048
#define MAX_TXD         4096
#define MIN_TXD         64
```


So, here's my kernel config for your viewing pleasure:

```
include         GENERIC

ident           SHAPER

nomakeoptions   DEBUG

nooptions       COMPAT_FREEBSD4         # Compatible with FreeBSD4
nooptions       COMPAT_FREEBSD5         # Compatible with FreeBSD5
nooptions       COMPAT_FREEBSD6         # Compatible with FreeBSD6
options         COMPAT_FREEBSD7         # Compatible with FreeBSD7
nooptions       COMPAT_FREEBSD32        # Compatible with i386 binaries

nooptions       INET6                   # IPv6 communications protocols
options         ZERO_COPY_SOCKETS
# XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work
#options        DEVICE_POLLING
options         IPFIREWALL              #firewall
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by default
```

Here's /etc/sysctl.conf:

```
net.inet.ip.fw.verbose=0

kern.ipc.shmall=65536
kern.ipc.shmmax=268435456
kern.ipc.semmap=1024
kern.ipc.nmbclusters=111111

net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3??
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000

hw.intr_storm_threshold=9000
#dev.em.0.rx_processing_limit=-1 # device not used any more
```

Any tips? I'll be happy to try and add some more info upon request.


Thanks.


----------



## rihad (Feb 9, 2011)

Problem solved, I'm so embarrassed  The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf: 
	
	



```
dummynet_enable="YES"
```
 and loaded it [cmd=]kldload dummynet[/cmd] in order to do without a reboot. Works like a charm so far. Thanks to all!


----------



## DutchDaemon (Feb 9, 2011)

rihad, please use proper formatting on these forums. Thanks.


----------



## rihad (Feb 9, 2011)

Ok, no problem. You guys could put a link to these rules in a place that everyone could see, that would probably save you some time, too.


----------



## DutchDaemon (Feb 9, 2011)

The General forum (the first one you enter) has half a dozen topics at the top (Sticky posts) dealing with this. There's a FAQ link at the top bar on every page. My signature can also serve as a useful source of information, though I don't post that much, as you can see. In the two years you've been here I have probably posted the link to this information under several thousands of posts. So, other than tattooing it on your arm, I'd say the information is within arm's reach on every single page  Thanks for your cooperation.


----------

