# Freebsd igb issue with i211-at embedded nics. Why drops all rx packets?



## dubel1976 (Dec 2, 2019)

On our custom motherboard (based on Intel Atom C3538) there are four i211-at nics each on a separate PCIe port.
The problem is that all incoming packets drops on these network adapters under FreeBSD. I test that on FreeBSD 11,3, 12.1, 13.0.
On other operating systems this problem is not. I checked it on CentOS, Fedora, Ubuntu and Windows 10.
When sending a ping command from one of i211 embedded nics, I see an arp request from it, but apparently it does not receive a response. Ping failed with "ping: sendto: Host is down" (I ping our gateway). The output of the "netstat -i" command shows incriasing droped packets.I did not make any addition configurations to the igb driver. All default settings after installation. I compiled the latest intel igb driver 2.5.9 for FreeBSD 11.3- it did not help. 

On the same motherboard there are other network adapters - two Marvell x553 1GBit - they work fine (on ixgbe driver).

Advise how to find the reason for this behavior (Ping not worcking - drops all rx packets)? Maybe I can somehow debug the igb driver?
I am new to FreeBSD, so what info still needs to be provided for a more complete picture of what is happening?


----------



## Phishfry (Dec 2, 2019)

FreeBSD 12 updated the network driver infrastructure and some adapters were broken by the new iflib(9) frameworks.
I would suggest you try FreeBSD 11.3 and see if that works for you.
If that works you could file a bug report with details and hopefully it gets fixed.


----------



## Phishfry (Dec 2, 2019)

If you want to diagnose FreeBSD >12 that I would start by looking at interrupt storm statistics.
`vmstat -i`
If the numbers look high try and tame them with:
`sysctl hw.intr_storm_threshold=4000`

Is this a single CPU rig or dual CPU? There are some NUMA issues with Dual CPU's that may need tweaking.


----------



## dubel1976 (Dec 3, 2019)

Well. I installed 11.3 - the result has not changed. I tried *vmstat -i* and s*ysctl hw.intr_storm_threshold=4000 *same result.


```
root@ipc:/home/ipc/igb-2.5.9/src # vmsts -i

interrupt                          total       rate
irq16: uart0                        3323          3
cpu0:timer                         14767         13
cpu1:timer                          7015          6
cpu2:timer                          6841          6
cpu3:timer                          7899          7
irq264: igb0:que 0                   950          1
irq265: igb0:que 1                   957          1
irq266: igb0:link                      2          0
irq277: ahci0:ch1                   6609          6
irq284: xhci0                       2135          2
Total                              50498         44
root@ipc:/home/ipc/igb-2.5.9/src # sysctl hw.intr_storm_threshold=4000
```

Here is what gives netstat -iW


```
root@ipc:~ # netstat -iW

Name    Mtu Network                  Address                                    Ipkts Ierrs Idrop    Opkts Oerrs  Coll
igb0   1500 <Link#1>                 70:b3:d5:f2:f0:a0                            142     0   630       72     0     0
igb0      - 192.168.0.0/24           192.168.0.234                                  0     -     -       78     -     -
igb1*  1500 <Link#2>                 70:b3:d5:f2:f0:a1                              0     0     0        0     0     0
igb2*  1500 <Link#3>                 70:b3:d5:f2:f0:a2                              0     0     0        0     0     0
igb3*  1500 <Link#4>                 70:b3:d5:f2:f0:a3                              0     0     0        0     0     0
ix0*   1500 <Link#5>                 70:b3:d5:f2:f0:a4                              0     0     0        0     0     0
ix1*   1500 <Link#6>                 70:b3:d5:f2:f0:a5                              0     0     0        0     0     0
ix2*   1500 <Link#7>                 70:b3:d5:f2:f0:a6                              0     0     0        0     0     0
ix3*   1500 <Link#8>                 70:b3:d5:f2:f0:a7                              0     0     0        0     0     0
lo0   16384 <Link#9>                 lo0                                            0     0     0        0     0     0
lo0       - localhost                localhost                                      0     -     -        0     -     -
lo0       - fe80::%lo0/64            fe80::1%lo0                                    0     -     -        0     -     -
lo0       - your-net                 localhost                                      0     -     -        0     -     -
```

We have not physically implemented the Wol function. Can it somehow affect?
How to find out which driver version is loaded? Modinfo command not working... After reboot, the old driver loads.


----------



## dubel1976 (Dec 3, 2019)

Forgot - motherboard has single CPU Intel Atom C3538 with four cores 2.1 GHz.


----------



## VladiBG (Dec 3, 2019)

The forum is used mostly for the end users. Try to contact some of the developers using the mailing list and if you can provide a test motherboard to them this will help for diagnose the problem.


----------



## Phishfry (Dec 4, 2019)

dubel1976 said:


> How to find out which driver version is loaded?


Use the source Luke.
usr/src/sys/dev/e1000/README

You can also load the newest drivers from ports.
net/intel-em-kmod/
net/intel-ix-kmod

You might want to try disabling some network card features like LRO, TSO and others.


----------



## SirDice (Dec 5, 2019)

dubel1976 said:


> I see an arp request from it, but apparently it does not receive a response.


If there's nothing responding to the ARP request then something else might be faulty. Can you see the ARP request arrive on the intended machine? And does it respond to it? Are you receiving the response? To make it a little easier to diagnose make sure to test with a machine in the same network subnet, so there's no router in between.


----------



## dubel1976 (Dec 12, 2019)

SirDice said:


> If there's nothing responding to the ARP request then something else might be faulty. Can you see the ARP request arrive on the intended machine? And does it respond to it? Are you receiving the response? To make it a little easier to diagnose make sure to test with a machine in the same network subnet, so there's no router in between.


Sorry for the long reaction. Apparently I did not configure the notification.Yes, I see ARP requests from i211 then ping gateway or other computer from it, but I don’t see any response packets. Moreover, by running the tcpdump on i211, I can see no packages at all. And there are plenty of them broadcast, multicast and so on. Once again, I note that everything works fine on other operating systems (CentOS, Fedora, Debian, Ubuntu). But it is very necessary that it would work specifically for FreeBSD. How to localize this problem. Enable extended logs or something like that? I attached boting log freebsd_putty.txt at top.


----------

