# "No IRQs left" Mellanox MT26448 (ConnectX-2 10Gb enet) on Supermicro X9SRL (FreeBSD 11.0-RELEASE)



## sawdustdood (Feb 5, 2017)

Howdy!  I'm having a heck of a time getting a pair of ConnectX-2 adapters working on my system.  I have tried loadable modules as well as a custom kernel with the modules/options built-in with the same result.  The system is generally very well behaved except for this problem.    I am simply out of ideas at this point in time (please excuse my first post).

When loading the kernel modules, it appears they load and the cards are recognized, but I get this "No IRQs left, device can't be started." madness.  This is using the Mellanox 2.1.6 drivers which come with 11.0-RELEASE and GENERIC kernel.

I have included an example of what happens with the module load, the lscpi, vmstat -ai showing interrupts, and attached kldstat showing all compiled-in and loaded modules and dmesg.  I am happy to post more, but I don't know what else would be helpful.


```
root@monstro:/ # kldload mlxen
root@monstro:/ # dmesg | tail
[IMG]https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif[/IMG]

mlx4_core0: <mlx4_core> mem 0xfb700000-0xfb7fffff,0xf3800000-0xf3ffffff irq 40 at device 0.0 on pci6
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb  3 2017)

mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
mlx4_core0: <mlx4_core> mem 0xfb300000-0xfb3fffff,0xf3000000-0xf37fffff irq 16 at device 0.0 on pci10
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb  3 2017)

mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
```



```
root@monstro:/ # lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1c.6 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 7 (rev b6)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
01:00.0 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (A-Segment Bridge) (rev 09)
01:00.2 PCI bridge: Intel Corporation 41210 [Lanai] Serial to Parallel PCI Bridge (B-Segment Bridge) (rev 09)
02:04.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
03:05.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5706 Gigabit Ethernet (rev 02)
05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
06:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
08:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
0a:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0c:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0d:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
```



```
root@monstro:/ # vmstat -ia
interrupt                          total       rate
???                                    0          0
irq1:                                  0          0
stray irq1                             0          0
irq0: attimer0                         0          0
stray irq0                             0          0
irq3:                                  0          0
stray irq3                             0          0
irq4:                                  0          0
stray irq4                             0          0
irq5:                                  0          0
stray irq5                             0          0
irq6:                                  0          0
stray irq6                             0          0
irq7:                                  0          0
stray irq7                             0          0
irq8: atrtc0                           0          0
stray irq8                             0          0
irq9: acpi0                            2          0
stray irq9                             0          0
irq10: uart2                           0          0
stray irq10                            0          0
irq11:                                 0          0
stray irq11                            0          0
irq12:                                 0          0
stray irq12                            0          0
irq13:                                 0          0
stray irq13                            0          0
irq14:                                 0          0
stray irq14                            0          0
irq15:                                 0          0
stray irq15                            0          0
irq16: ehci0                     9526835         97
stray irq16                            0          0
irq17:                                 0          0
stray irq17                            0          0
irq18:                                 0          0
stray irq18                            0          0
irq19:                                 0          0
stray irq19                            0          0
irq20:                                 0          0
stray irq20                            0          0
irq21:                                 0          0
stray irq21                            0          0
irq22:                                 0          0
stray irq22                            0          0
irq23: ehci1                      201906          2
stray irq23                            0          0
irq24:                                 0          0
stray irq24                            0          0
irq25:                                 0          0
stray irq25                            0          0
irq26: bce0                      9032712         92
stray irq26                            0          0
irq27:                                 0          0
stray irq27                            0          0
irq28: bce1                     10612876        108
stray irq28                            0          0
irq29:                                 0          0
stray irq29                            0          0
irq30:                                 0          0
stray irq30                            0          0
irq31:                                 0          0
stray irq31                            0          0
irq32:                                 0          0
stray irq32                            0          0
irq33:                                 0          0
stray irq33                            0          0
irq34:                                 0          0
stray irq34                            0          0
irq35:                                 0          0
stray irq35                            0          0
irq36:                                 0          0
stray irq36                            0          0
irq37:                                 0          0
stray irq37                            0          0
irq38:                                 0          0
stray irq38                            0          0
irq39:                                 0          0
stray irq39                            0          0
irq40:                                 0          0
stray irq40                            0          0
irq41:                                 0          0
stray irq41                            0          0
irq42:                                 0          0
stray irq42                            0          0
irq43:                                 0          0
stray irq43                            0          0
irq44:                                 0          0
stray irq44                            0          0
irq45:                                 0          0
stray irq45                            0          0
irq46:                                 0          0
stray irq46                            0          0
irq47:                                 0          0
stray irq47                            0          0
cpu0:timer                      33848633        343
irq256: hpet0:t0                       0          0
stray irq256                           0          0
irq257: hpet0:t1                       0          0
stray irq257                           0          0
irq258: hpet0:t2                       0          0
stray irq258                           0          0
irq259: hpet0:t3                       0          0
stray irq259                           0          0
irq260: hpet0:t4                       0          0
stray irq260                           0          0
irq261: hpet0:t5                       0          0
stray irq261                           0          0
irq262: hpet0:t6                       0          0
stray irq262                           0          0
irq263: hpet0:t7                       0          0
stray irq263                           0          0
irq264: xhci0                          0          0
stray irq264                           0          0
irq265: mpr0                     8147997         83
stray irq265                           0          0
irq266: em0                            0          0
stray irq266                           0          0
irq267: em1                            0          0
stray irq267                           0          0
irq268: ahci0                         27          0
stray irq268                           0          0
cpu1:timer                      11680909        118
cpu4:timer                      12066595        122
cpu7:timer                      11196384        113
cpu6:timer                      13054931        132
cpu5:timer                      11188758        113
cpu2:timer                      11319839        115
cpu11:timer                     11077046        112
cpu9:timer                      11335830        115
cpu8:timer                      11665891        118
cpu3:timer                      11244335        114
cpu10:timer                     13033996        132
Total                          200235502       2030
```

Any help or suggestions would be appreciated.

Thanks,
Ted


----------



## sawdustdood (Feb 6, 2017)

Tried to compile the 2.1 and 2.1.5 drivers with FBSD 11 kernel source and there appear to be a number of conflicts with kernel headers, etc. so this doesn't seem incredibly viable.  Friend has same card working in 10.3 with 2.1 drivers.


----------



## sawdustdood (Jul 30, 2017)

Moved from (old) Mellanox to (old) Chelsio and far fewer problems.  There seems to be better support for old Chelsio cards.


----------

