# MSI/MSI-X on Intel em NIC



## wrkilu (Nov 15, 2011)

Hi,

We have a border router with 5 em nics. OS is FreeBSD 8.2 x64. 

```
# top -PSH
last pid:  6097;  load averages:  1.46,  1.45,  1.48                                                    
                                                      up 31+13:32:57  14:02:32
145 processes: 11 running, 111 sleeping, 23 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 30.3% interrupt, 69.7% idle
CPU 1:  0.4% user,  0.0% nice,  0.4% system,  4.5% interrupt, 94.7% idle
CPU 2:  0.0% user,  0.0% nice,  3.8% system, 10.5% interrupt, 85.7% idle
CPU 3:  0.0% user,  0.0% nice, 46.8% system,  0.4% interrupt, 52.8% idle
CPU 4:  0.0% user,  0.0% nice, 29.6% system,  0.0% interrupt, 70.4% idle
CPU 5:  0.4% user,  0.0% nice, 14.7% system,  0.0% interrupt, 85.0% idle
CPU 6:  0.0% user,  0.0% nice, 64.8% system,  0.0% interrupt, 35.2% idle
CPU 7:  0.0% user,  0.0% nice,  0.0% system,  4.9% interrupt, 95.1% idle
Mem: 374M Active, 2576M Inact, 782M Wired, 31M Cache, 412M Buf, 91M Free
Swap: 10G Total, 10G Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root       171 ki31     0K   128K CPU7    7 715.5H 100.00% {idle: cpu7}
   11 root       171 ki31     0K   128K CPU1    1 719.5H 96.53% {idle: cpu1}
   11 root       171 ki31     0K   128K RUN     5 680.7H 86.87% {idle: cpu5}
   11 root       171 ki31     0K   128K CPU2    2 708.4H 78.71% {idle: cpu2}
   11 root       171 ki31     0K   128K CPU4    4 510.5H 72.71% {idle: cpu4}
   11 root       171 ki31     0K   128K CPU0    0 549.6H 71.92% {idle: cpu0}
    0 root       -68    0     0K   208K -       6 380.0H 65.97% {em4 taskq}
   11 root       171 ki31     0K   128K CPU3    3 466.1H 57.86% {idle: cpu3}
    0 root       -68    0     0K   208K CPU3    3 290.3H 45.26% {em1 taskq}
   11 root       171 ki31     0K   128K CPU6    6 384.3H 39.40% {idle: cpu6}
    0 root       -68    0     0K   208K CPU4    4 245.6H 33.84% {em2 taskq}
   12 root       -68    -     0K   368K WAIT    0 204.1H 31.64% {irq256: em0:rx 0}
   12 root       -44    -     0K   368K WAIT    3  81.7H 25.49% {swi1: netisr 0}
    0 root       -68    0     0K   208K -       5  53.9H 14.94% {em3 taskq}
   12 root       -68    -     0K   368K WAIT    1 506:54  1.22% {irq257: em0:tx 0}
    0 root       -68    0     0K   208K -       7  26.5H  0.78% {dummynet}
14434 root        44    0 26984K  5800K select  3 279:09  0.63% snmpd
   12 root       -32    -     0K   368K WAIT    1 243:50  0.00% {swi4: clock}
    0 root       -68    0     0K   208K -       4 139:00  0.00% {em0 txq}
   13 root        44    -     0K    16K -       4  85:22  0.00% yarrow
    0 root       -68    0     0K   208K -       1  64:54  0.00% {em0 rxq}
```


Hot to turn on MSI on these cards? How to check is it enabled?


----------



## wblock@ (Nov 15, 2011)

Check /var/log/messages:

```
em0: Using MSIX interrupts with 3 vectors
```

I thought there was a sysctl to disable MSI, but looking through the output of sysctl -a I don't see one for em(4).


----------



## phoenix (Nov 15, 2011)

It all depends on the version of the em(4) driver, the version of the NIC chipset, and the version of FreeBSD in use.  For example, on my 9.0-RC2 box, I have the following sysctl (there's no em-based NIC in this box, though):

```
hw.em.enable_msix: 1
```

On my 8.2-STABLE (r224019) box with this NIC:

```
em0@pci0:9:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82571EB Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
```
I don't have that sysctl.


----------



## wrkilu (Nov 16, 2011)

Well, we've found the reason, Only em0 on our router supports MSI-X and only she has few interrupts on few cores. In Freebsd 8.2 that feature is enabled in default.


```
# pciconv -lcv
em0@pci0:1:0:0: class=0x020000 card=0x34da8086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)
[B]    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled[/B]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected


em1@pci0:2:0:0: class=0x020000 card=0x10828086 chip=0x107d8086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 PT'
    class      = network
    subclass   = ethernet
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 256(256) link x1(x1)
```

regards


----------

