# igb & interrupt storm detected



## ikyb (Dec 11, 2009)

After upgrade system i have problem with igb driver. Old network interface (em) forward 115-130k pakets on new network adapter igb (Intel 82576) maximum 100-110k packets. 

dmesg show 

```
...
interrupt storm detected on "irq257:"; throttling interrupt source
...

gw7# vmstat -i
interrupt                          total       rate
irq18: ehci0 uhci5                     2          0
irq19: uhci2 uhci*                192701          2
irq21: uhci1                         765          0
irq23: uhci3 ehci1                     1          0
cpu0: timer                    135336795       1981
irq256: igb0                   190500970       2789
irq257: igb0                    83395261       1221
irq258: igb0                           3          0
cpu5: timer                    135336542       1981
cpu11: timer                   135425550       1983
cpu7: timer                    135336538       1981
cpu15: timer                   135425487       1983
cpu2: timer                    135336605       1981
cpu9: timer                    135425559       1983
cpu1: timer                    135336542       1981
cpu8: timer                    135425558       1983
cpu4: timer                    135336542       1981
cpu10: timer                   135425589       1983
cpu3: timer                    135336617       1981
cpu12: timer                   135425561       1983
cpu6: timer                    135336537       1981
cpu14: timer                   135425527       1983
cpu13: timer                   135425561       1983
Total                         2440186813      35731
```

`gw7# netstat -w1 -i igb0`

```
input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
     53902     0   37779755      52207     0   32702384     0
     55127     0   38909372      53402     0   33609417     0
     55576     0   38466648      53549     0   33645440     0
```

`gw7# cat /boot/loader.conf`

```
hw.igb.rxd=2048
hw.igb.txd=2048
```

`set dev.igb.0.enable_lro=0` not help

`gw7# cat /etc/sysctl.conf`

```
vfs.read_max=128
net.inet.ip.fw.one_pass=0
net.inet.ip.fastforwarding=1
net.inet.icmp.icmplim=1000
net.inet.ip.dummynet.io_fast=1
```



```
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
        ether 00:30:48:ce:fe:6c
        inet xx.yy.16.22 netmask 0xffffffc0 broadcast xx.yy.16.63
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
```

Old server 
2*Xenon E5520 dual port network adapter Intel 82546 (em)
OS Freebsd 8.0-STABLE amd64

New server
2*Xeon X5550 dual port network adapter Intel 82576 (igb)
OS Freebsd 8.0-STABLE amd64

At the time of writing a letter to the load about ~40% of the maximum. 

P.S. Server role nat+shape. 

What can I do to increase performance ?


----------



## m87carlson (Dec 11, 2009)

Are you using natd or pf's nat implementation?

I was told that pf's nat implementation works in the kernel, while natd is a userland application, which means the kernel has to copy each packet in and out of the kernel.

Otherwise, I'm interested in what you come up with, I'll be working with a similar server setup soon.


----------



## ikyb (Dec 12, 2009)

m87carlson said:
			
		

> Are you using natd or pf's nat implementation?
> 
> I was told that pf's nat implementation works in the kernel, while natd is a userland application, which means the kernel has to copy each packet in and out of the kernel.
> 
> Otherwise, I'm interested in what you come up with, I'll be working with a similar server setup soon.



I use ipfw nat (IPFIREWALL_NAT) and DUMMYNET, this design works well (previously used ng_nat). pf nat I have not tried, but I do not think that would be a big difference.

driver downloaded from intel v.1.7.4 leads to the collapse of systems 

I expect the problem is that the igb driver does not support polling, now ordered a network card on the old chip that supports polling.

P.S. In a network many reports of problems with igb driver but solution is not found ...


----------



## willian_pires (Sep 1, 2010)

*Try balance over cpus*

Execute the follow steps:

`cpuset -l 3 -x 256`
`cpuset -l 4 -x 257`
`cpuset -l 5 -x 258`

Try recompile your kernel with HZ=2000 or above value

Att
Willian Pires Souza


----------

