# Difficulties with SR-IOV on Chelsio T4 cards (driver can’t get interrupts)



## arader (May 3, 2019)

Hi all,

I’m finally upgrading to 10Gbe with a pair of Chelsio T420-CR cards and one of my goals is to get off of `bridge` and `epair` devices for my jails and instead use VF devices from SR-IOV.

However, when trying to enable these virtual devices the drivers don’t attach. What I have so far:

12.0-RELEASE p3
Intel Xeon 1245v3

`cat /boot/loader.conf`

```
zfs_load="YES"
t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
if_cxgbev_load="YES"
t4_tom_load="YES"
```
Then I have an `iovctl` config file for my device:
`cat /etc/iovctl.cxgbe0.conf`

```
PF {
    device : "cxgbe0";
    num_vfs : 4;
}
```
I can confirm that this enables the VF portions of the card:
`doas pciconf -lvc`

```
t4iov0@pci0:1:0:0:      class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI disabled
                     4 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xef000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xef004000, size 8192, enabled
    ecap 000e[204] = ARI 1
```
However, when I look at `ifconfig` I don’t see any new interfaces. Sure enough, if I look at `dmesg` I see 4 entries of the following:


```
t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci1
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6
```
So it looks like the drivers can’t get any interrupts for the interfaces. Looking at other entries in `pciconf` I do see several using MSI and MSI-X, so it’s not totally broken, but it might be interesting that the Chelsio devices don’t have MSI/MSI-X enabled...

So, anyone have any thoughts? MSI-X appears to be an automatic thing on FreeBSD, should I be tweaking the BIOS or something?

thanks!


----------



## Phishfry (May 3, 2019)

I don't know about your interrupt errors.

So I have 4 SR-IOV devices:

```
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov1@pci0:4:0:1:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov2@pci0:4:0:2:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4iov3@pci0:4:0:3:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t4nex0@pci0:4:0:4:    class=0x020000 card=0x00001425 chip=0x44011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
```
I have only been able to get one interface working this way with VF's.
I ppt passthru one of the t4iov devices and t4nex0 in the loader.
With that I get a virtualized chelsio interface inside a bhyve VM.
It seems to take the combination of the two devices for some reason.

There is a totally separate second way. That may take another post...
/boot/loader.conf
#hw.cxgbe.num_vis=2


----------



## Phishfry (May 3, 2019)

Seeing how you are on a E3 Xeon you might need this /boot/loader.conf tweak
#hw.vmm.force_iommu=1






						229852 – bhyve: IOMMU (Intel VTd) PCI passthrough attempt locks up some systems
					






					bugs.freebsd.org


----------



## Phishfry (May 3, 2019)

Here is what I used so far beside the iovctl manual.:


			Testing VF/PF code
		



			svn commit: r302110 - head/sys/dev/cxgbe


----------



## Phishfry (May 3, 2019)

I see you are using a setting that is unknown to me.
t4_tom_load="YES"

That is probably what I am missing.

My loader only has the basics:
t4fw_cfg_load="YES"
if_cxgbe_load="YES"

I see that setting here too:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216539


----------



## arader (May 3, 2019)

Thanks Phishfry, the t4_tom module is needed for enabling TOE on these cards, since I'm not using it right now I removed that from my imports. Still the same issue.

I snooped around in my BIOS and couldn't find anything obvious, VT-d is listed as enabled. I couldn't find any interrupt related options, which makes sense (the days of IRQ numbers are long behind us) so I'm at a bit of a loss.

Phishfry can you do me a favor and post `pciconf -lcv` for your Chelsio devices? I'm curious if they're using MSI/MSI-X in your setup. I have the same t4iov/t4nex0 devices as you.

Frankly I'm a little disappointed in the documentation around SR-IOV and FreeBSD - this is the first time I'm having to really scrape the edges of the internet to get any details on how to set this up.


----------



## Phishfry (May 3, 2019)

Here you go:

```
root@virt:~ # pciconf -lbcev t4iov0
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    ecap 000e[204] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Receiver Error
                 Advisory Non-Fatal Error
```

Intresting that you show this interface:


arader said:


> t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci1


So this is in the VM?
I have never seen that interface. I did have vcxgbe0 with the second method.
Like I say if I pass thru both t4iov and t4nex0 I get one virtual interface but it uses cxgbe moniker.
So it looks like I am not doing it right. I need to add "if_cxgbev_load="YES"" to my setup.
Yes instructions would be nice. I am using two i-350 quad Intel NIC for now.
I did read where t4_tom is TOE so that is irrelevant. Offloading is interesting but not needed with my load.


```
root@virt:~ # sysctl -a | egrep -i 'hw.machine|hw.ncpu|hw.model'
hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
hw.ncpu: 48
hw.machine_arch: amd64
```


----------



## Phishfry (May 3, 2019)

Arg I forgot to uncomment out this line in /etc/rc.conf
iovctl_files="/etc/cxgbe0.conf"
I have been down this road already...Hopefully I get further this time.
That is why the pciconf above shows no sr-iov.


----------



## Phishfry (May 4, 2019)

Here is relevant dmesg for my card showing interrupts.

```
cxgbe0: <port 0> numa-domain 0 on t4nex0
cxgbe0: Ethernet address: 0
cxgbe0: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
cxgbe1: <port 1> numa-domain 0 on t4nex0
cxgbe1: Ethernet address: 0
cxgbe1: 16 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
t4nex0: PCIe gen2 x1, 2 ports, 22 MSI-X interrupts, 71 eq, 21 iq
```


----------



## Phishfry (May 4, 2019)

I have finally got somewhere. I have 4 VF's

```
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled
                     4 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xc7000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xc7004000, size 8192, enabled
    ecap 000e[204] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Receiver Error
                 Advisory Non-Fatal Error
```


----------



## Phishfry (May 4, 2019)

And jacked up to full blast we get 16 VF's. That is only one interface. cxgbe0. So 32 VF's possible with a dual port card.

```
t4iov0@pci0:4:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xc7380000, size 262144, enabled
    bar   [20] = type Memory, range 64, base 0xc744c000, size 8192, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x1000]
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR NS
                 link x1(x8) speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 2 corrected
    ecap 0008[138] = MFVC 1
    ecap 0003[1b8] = Serial 1 0000000000000000
    ecap 0010[1c4] = SR-IOV 1 IOV enabled, Memory Space enabled, ARI enabled
                     16 VFs configured out of 16 supported
                     First VF RID Offset 0x0008, VF RID Stride 0x0004
                     VF Device ID 0x4801
                     Page Sizes: 4096 (enabled)
    iov bar  [1e8] = type Memory, range 64, base 0xc7000000, size 4096, enabled
    iov bar  [1f8] = type Memory, range 64, base 0xc7010000, size 8192, enabled
    ecap 000e[204] = ARI 1
```


----------



## arader (May 4, 2019)

From my reading, if you have “passthrough” set to true in the iovctl conf file you get a ‘ppt’ pci device that can be passed to a VM. If it’s false then you can directly attach in the host and create an interface for jails.

The t4vf0 comes from the cxgbe*v* driver, after loading that i would expect to see virtual interfaces show in the host.

Looking through the cxgbev driver sys/dev/cxgbe/t4_vf.c I can definitely see that my interrupt error in dmesg is why i’m not seeing any interfaces. That gets logged in the failure path and nothing gets attached. If the driver were able to get some interrupts it would have moved on to attaching the devices.

I just noticed that pciconf shows my t4nex0 device as using MSI-X so it is working. I’m going to try flashing an updated BIOS tomorrow and see if that changes anything. Something is causing the kernel to report 0 available interrupts...


----------



## Phishfry (May 4, 2019)

arader said:


> If it’s false then you can directly attach in the host and create an interface for jails.


This setting of 'false' was crashing my machine. It is odd because it was in my config from my earlier attempt.
I am sure I have something set wrong. For now it is all passthru. None for the host.


----------



## arader (May 4, 2019)

So I just tried setting passthrough to true and i can get a ‘ppt’ device just fine. I’m beginning to suspect that “non-passthrough” just isn’t supported by these drivers (yet?) and I’ll have to just keep using bridge+epair in my jails.

At least I can pass through the VF for my VMs!


----------



## arader (May 6, 2019)

Updated the BIOS on my Supermicro X10SAT and still no luck, I can't seem to get any interfaces attached for non-passthrough VFs on the T420-CR.

Additionally, if I create multiple VFs and set them all to "passthrough: true" I still only end up with 1 ppt device. I would assume I should end up with a ppt for each VF that I can pass through?

Man, I was told Chelsio was well supported on FreeBSD but I'm really not getting that vibe here...


----------



## Nyakov (Nov 17, 2020)

Hi.
I have same problem.
FreeBSD 12.2.
Chelsio T540-CR
Message I get:
`t5vf0: <Chelsio T540-CR VF> at device 1.0 on pci1
t5vf0: failed to find a usable interrupt type. allowed=7, msi-x=0, msi=0, intx=1device_attach: t5vf0 attach returned 6`


----------



## Nyakov (Nov 17, 2020)

Ok. This is what I figured out.

1. Virtual interfaces for vnet in Jails you can obtain via hw.cxgbe.num_vis tunable in /boot/loader.conf
How to control number of vis for separate physical port remains mystery?

2. I do not see this virtual interfaces in PCI devices (pciconf -l)

3. PCI devices for bhyve passthru can be obtained by iovctl config. But they cannot be used by host system.


----------



## UrsusDominatus (Jan 25, 2021)

Hi. If anyone still interested in this topic, I have an information about this issue. Recently I've experimenting with SR-IOV function on Chelsio T4 (cxgbe driver) and Intel 82599ES (ixgbe driver). I have issues with both, but here we talk about Chelsio. I have Supermicro X11SSM-F motherboard with Chelsio with T420-CR in it. First I have no issue mentioned here "failed to find a usable interrupt type". After some time I need to rearrange expansion cards in my server and moved NIC to different PCI-E slot, and afer reboot exactly this issue appear:


```
t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci2                               
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1
```

After some more experimenting with physically moving NIC back and forth in MB slots, turned out, that in PCI-E x4 (in x8) slot everything works fine (I have usable cxgbev0 VF's that can be configured and used in jails). But if NIC in PCI-E x8 slots "failed to find usable interrupt type" issue is appear.

Here some logs for clarification.
"Error" state:

```
t4nex0: <Chelsio T420-CR> mem 0xdf280000-0xdf2bffff,0xde800000-0xdeffffff,0xdf3c4000-0xdf3c5fff irq 17 at device 0.4 on pci2
t4nex0: PCIe gen2 x8, 2 ports, 14 MSI-X interrupts, 30 eq, 13 iq                                                            
cxgbe0: <port 0> on t4nex0                                                                                                  
cxgbe0: Ethernet address: 00:07:43:14:d7:40                                                                                 
cxgbe0: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)                                                                        
cxgbe1: <port 1> on t4nex0                                                                                                  
cxgbe1: Ethernet address: 00:07:43:14:d7:48                                                                                 
cxgbe1: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)                                                                        
t4vf0: <Chelsio T420-CR VF> at device 1.0 on pci2                                                                           
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6     
t4vf0: <Chelsio T420-CR VF> at device 1.4 on pci2                                                                           
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6     
t4vf0: <Chelsio T420-CR VF> at device 2.0 on pci2                                                                           
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6     
t4vf0: <Chelsio T420-CR VF> at device 2.4 on pci2                                                                           
t4vf0: failed to find a usable interrupt type.  allowed=7, msi-x=0, msi=0, intx=1device_attach: t4vf0 attach returned 6
```

"No error" state:

```
t4nex0: <Chelsio T420-CR> mem 0xdf280000-0xdf2bffff,0xde800000-0xdeffffff,0xdf3c4000-0xdf3c5fff irq 16 at device 0.4 on pci3
t4nex0: PCIe gen2 x4, 2 ports, 14 MSI-X interrupts, 30 eq, 13 iq                                                            
cxgbe0: <port 0> on t4nex0                                                                                                  
cxgbe0: Ethernet address: 00:07:43:14:d7:40                                                                                 
cxgbe0: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)                                                                        
cxgbe1: <port 1> on t4nex0                                                                                                  
cxgbe1: Ethernet address: 00:07:43:14:d7:48                                                                                 
cxgbe1: 4 txq, 4 rxq (NIC); 4 txq (TOE), 2 rxq (TOE)                                                                        
t4vf0: <Chelsio T420-CR VF> at device 0.8 on pci3                                                                           
cxgbev0: <port 0> on t4vf0                                                                                                  
cxgbev0: Ethernet address: 06:44:14:d7:40:00                                                                                
cxgbev0: 2 txq, 1 rxq (NIC)                                                                                                 
t4vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq                                                                              
t4vf1: <Chelsio T420-CR VF> at device 0.12 on pci3                                                                          
cxgbev1: <port 0> on t4vf1                                                                                                  
cxgbev1: Ethernet address: 06:44:14:d7:40:01                                                                                
cxgbev1: 2 txq, 1 rxq (NIC)                                                                                                 
t4vf1: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq                                                                              
t4vf2: <Chelsio T420-CR VF> at device 0.16 on pci3                                                                          
cxgbev2: <port 0> on t4vf2                                                                                                  
cxgbev2: Ethernet address: 06:44:14:d7:40:02                                                                                
cxgbev2: 2 txq, 1 rxq (NIC)                                                                                                 
t4vf2: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq                                                                              
t4vf3: <Chelsio T420-CR VF> at device 0.20 on pci3                                                                          
cxgbev3: <port 0> on t4vf3                                                                                                  
cxgbev3: Ethernet address: 06:44:14:d7:40:03                                                                                
cxgbev3: 2 txq, 1 rxq (NIC)                                                                                                 
t4vf3: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
```

My IOV config is very simple:

```
PF {
    device: cxgbe0;
    num_vfs: 4;
}
```


----------



## arader (Feb 5, 2021)

that's interesting! any idea if the slots that work connect directly to the CPU vs the PCH?

I haven't any any luck trying different slots on my X10-SAT motherboard. I will say with the same exact card and drivers on a M11SDV-8CT-LN4F (amd epyc mini-ITX board) allows SR-IOV to work just fine. I'm leaning towards it being either a motherboard or BIOS issue with the X10-SAT.


----------

