# SR-IOV (Chelsio) error in guest



## pos (May 4, 2019)

The Chelsio T520-BT card work if inserted into a linux host. Now it is placed in FreeBSD 12.0p3 RELEASE host and given out to a CentOS 7.6 guest with pass through. Then I can see this in the guest....

 3.466926] cxgb4vf: Chelsio T4/T5/T6 Virtual Function (VF) Network Driver - version 2.0.0-ko
[    3.466995] cxgb4vf 0000:00:06.0: enabling device (0004 -> 0006)
[    3.488970] cxgb4vf 0000:00:06.0: bad SGE FL buffer sizes [2048, 4096]
[    3.489005] cxgb4vf 0000:00:06.0: unable to use adapter parameters: err=-22
[    3.489112] cxgb4vf: probe of 0000:00:06.0 failed with error -22


So I cannot make the passed though card to work in the guest. Maybe a linux bug. But this card worked with SR-IOV pci pass through from Linux host to a linux guest. But not if the host is FreeBSD. So therefor I wonder....


Any clues very much appreciated

/Peo


----------



## Phishfry (May 6, 2019)

Well i finally got an interface up in a VM.

```
t4vf0@pci0:0:4:0:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```


```
root@freebsd2:~ # ifconfig -v cxgbev0
cxgbev0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether 06:44:14:f9:00:01
    inet 192.168.1.134 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet 10Gbase-SR <full-duplex>
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

So on FreeBSD 11.2 PF and VF work on Chelsio.
dmesg

```
t4vf0: <Chelsio T420-CR VF> mem 0xc0060000-0xc0060fff,0xc0062000-0xc0063fff at device 4.0 on pci0
cxgbev0: <port 0> on t4vf0
cxgbev0: Ethernet address: 06:44:14:f9:00:01
cxgbev0: 2 txq, 1 rxq (NIC)
t4vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
```


----------



## Phishfry (May 6, 2019)

Host settings:
/etc/rc.conf

```
ifconfig_cxgbe0="DHCP"
ifconfig_cxgbe1="DHCP"
iovctl_files=/etc/iovctl.conf
```
/boot/loader.conf

```
t4fw_cfg_load="YES"
if_cxgbe_load="YES"
pptdevs="4/0/1";###<<<The special sauce<< Not the first PCI address(t4iov0) but second(was t4iov1).
```
/etc/iovctl.conf

```
PF {device:"cxgbe0";num_vfs:2;}
VF-0 {passthrough:false;}
VF-1 {passthrough:true;}
```
Here is pciconf for the new devices.

```
none173@pci0:4:0:8:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt5@pci0:4:0:12:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```
So for my bhyve script I passthru 4/0/12
The other device is on the host. If I enable if_cxgbev="YES" in loader.conf  it crashes the host.
But the device is present in pciconf.
So maybe I will figure that out one day. Maybe an issue having both if_cxgbe and if_cxgbev enabled on the host.
if_cxgbe must be enabled on the host.
.
VM settings:
/etc/rc.conf

```
ifconfig_cxgbev0="DHCP"
```
/boot/loader.conf

```
if_cxgbev_load="YES"
```

On the host:
Once you add the ppt device for the second PCI address(for me 4/0/1) new devices will appear at botton of pciconf -lv
For me the new devices started at 4/0/8 for the Chelsio VF interfaces.
I stumbled in this by accident. I meant to add the first address for t4iov0 which is 4/0/0 but added 4/0/1 by accident.
It was actually a good thing.
Now what was t4iov1 on 4/0/1 is now t4iov1 on 4/0/2.


----------



## Phishfry (May 6, 2019)

I also noticed a special relationship with a file named /etc/iovctl.conf. It will mess with a iovctl setup.
If I have a /etc/cxgbe0.conf and have a /etc/iovctl.conf iov will fail to work.


----------



## Phishfry (May 6, 2019)

I increased my VF's  to 15 and passthru to bhyve 3 interfaces. 4/0/12, 4/0/16 and 4/0/20

```
cxgbev0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:01
        inet 192.168.1.134 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxgbev1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:02
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxgbev2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
        ether 06:44:14:f9:00:03
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
Need to setup my /etc/rc.conf to get my additional DHCP leases.


----------



## Phishfry (May 6, 2019)

I next want to setup a /etc/cxgbe1.conf to see if I can get IOV working on both interfaces for a total of 32 VF's.
The /etc/rc.conf setting of [iovctl_files=""] can have multiple conf files.


----------



## pos (May 6, 2019)

> The other device is on the host. If I enable if_cxgbev="YES" in loader.conf it crashes the host.
> But the device is present in pciconf.
> So maybe I will figure that out one day. Maybe an issue having both if_cxgbe and if_cxgbev enabled on the host.
> if_cxgbe must be enabled on the host.

I have both loaded. Otherwise you wont se VFs in the host. No crashes


> I next want to setup a /etc/cxgbe1.conf to see if I can get IOV working on both interfaces for a total of 32 VF's.
> The /etc/rc.conf setting of [iovctl_files=""] can have multiple conf files.

I have put my files in /etc/iov/


----------



## arader (May 6, 2019)

Phishfry, congrats, it sounds like your setup is mostly working! I wish this wasn't such a mysterious thing, SR-IOV is hardly new functionality...

Am I corrrect in my understanding that for every VF you're creating, you're seeing a new device appear in `pciconf`?

Also, I'm not sure how, but according to Chelsio's documentation it should be possible to get *128* VFs off of these cards: https://www.chelsio.com/nic/t4-unified-wire-adapters/t420-cr/


----------



## Phishfry (May 6, 2019)

arader said:


> for every VF you're creating, you're seeing a new device appear in  pciconf?


Yes exactly. They will not show up unless I have ppt entry for the 'second' Chelsio device by number.
For example if t4iov0 shows up on your computer as 12/0/0 then you would add a ppt entry for 12/0/1
Then reboot and see the VF's.

So my Chesio t4iov0 interface was 4:0:0, So I need to add 4:0:1 for passthru.
Then pciconf shows the PF's at the very end.
Note I am also passing through a quad gigabit adapter I use for each VM currently.


```
root@virt:~ # cat /boot/loader.conf
coretemp_load="YES"
pptdevs="4/0/1 132/0/0 132/0/1 133/0/0 133/0/1"
vmm_load="YES"
nmdm_load="YES"
t4fw_cfg_load="YES"
if_cxgbe_load="YES"
#if_cxgbev_load="YES"
#t4_tom_load="YES"
#hw.cxgbe.num_vis=2
#hw.vmm.force_iommu=1
```

These devices end up in pciconf after I reboot with ppt settings and iovctl properly configured.
So none@173 here is my passthrough:false VF. I may reroute this to a guest as it does me no good.
cxgbev consistently causes a kernel crash on my host when enabled. Machine does looping reboot. cxgbev works great in a VM.

```
none173@pci0:4:0:8:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt5@pci0:4:0:12:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt6@pci0:4:0:16:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt7@pci0:4:0:20:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt8@pci0:4:0:24:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt9@pci0:4:0:28:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt10@pci0:4:0:32:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt11@pci0:4:0:36:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt12@pci0:4:0:40:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt13@pci0:4:0:44:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt14@pci0:4:0:48:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt15@pci0:4:0:52:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt16@pci0:4:0:56:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt17@pci0:4:0:60:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt18@pci0:4:0:64:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt19@pci0:4:0:68:    class=0x020000 card=0x00001425 chip=0x48011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T420-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```


----------



## Phishfry (May 6, 2019)

This is as far as I get when trying to bring up the cxgbev interface manually on the host:

```
root@virt:~ # ifconfig cxgbev0
cxgbev0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 06:44:14:f9:00:00
    hwaddr 06:44:14:f9:00:00
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Fssh_packet_write_wait: Connection to 192.168.1.50 port 22: Broken pipe
```
Reboots after maybe 60 seconds.
Same if cxgbev added to /boot/loader.conf but requires USB stick to bring it back to life. fsck required for rescue mounting.


----------



## Phishfry (May 7, 2019)

I wanted to add that I am using 11.2-STABLE for my host (legacy BIOS) and my VM's are all EFI 12-RELEASE


arader said:


> according to Chelsio's documentation it should be possible to get *128* VFs off of these cards


I wonder if this is marketing or for real. Notice the term they use. 128 Virtual Functions.
Maybe they are counting each filter you use as a function. ie passthrough and mac-addr.

I tried to use my second interface, cxgbe1 as a iov device as well and have failed.
Even if I try and _only_ use cxgbe1 I get no VF's.


> root@virt:~ # cat /etc/cxgbe1.conf
> PF {device:"cxgbe1";num_vfs:4;}
> DEFAULT {passthrough:true;}


Only cxgbe0 is able to attach a node at /dev/iov/
Tried this:
`iovctl_files="/etc/cxgbe0.conf etc/cxgbe1.conf"`As well as manually:


> root@virt:~ # iovctl -f /etc/cxgbe1.conf -C
> iovctl: Could not open device '/dev/iov/cxgbe1': No such file or directory
> 
> root@virt:~ # iovctl -d cxgbe1 -S
> iovctl: Could not open device '/dev/iov/cxgbe1': No such file or directory



What is strange is there are 4 IOV devices in pciconf and I can only configure one for 16 VF, t4iov0.
t4iov0 is seemingly linked to cxgbe0. When I use iovctl it requires an interface node. Not a IOV node.
t4nex0 is the parent interface, but strangely it uses a PCI address of 4:0:4.
Usually a parent device uses the top level PCI address. In this case 4:0:0
If you look through the source there is interesting tidbits in the /usr/src/sys/dev/cxgbe/firmware directory.
t4fw_cfg.txt
Line 100 here alludes to t4nex0 PCI address of x:0:4


> # PF4 is the resource-rich PF that the bus/nexus driver attaches to.
> # It gets 32 MSI/128 MSI-X vectors.


----------



## openjohn (Aug 16, 2019)

I'm working through this with a T520-CR and figured out some of your confusion. Here's my pciconf output:


```
t5iov0@pci0:3:0:0:      class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov1@pci0:3:0:1:      class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov2@pci0:3:0:2:      class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov3@pci0:3:0:3:      class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5nex0@pci0:3:0:4:      class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
none3@pci0:3:0:5:       class=0x010000 card=0x00001425 chip=0x55011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Storage Controller'
    class      = mass storage
    subclass   = SCSI
none4@pci0:3:0:6:       class=0x0c0400 card=0x00001425 chip=0x56011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Storage Controller'
    class      = serial bus
    subclass   = Fibre Channel
```

Looking at the firmware documentation in the FreeBSD source tree, t5iov[0-3] appear to correspond to the PFs, that is, the physical ports (some Chelsio cards have 4 ports).

For me, both interfaces are listed in /dev/iov:

```
# ls -l /dev/iov
crw-------  1 root  wheel  0x31 Aug 16 15:05 cxl0
crw-------  1 root  wheel  0x32 Aug 16 15:05 cxl1
```

However, when I use pptdevs="3/0/1" for t5iov1, this masks /dev/iov/cxl1 (because the PF is put into pass-through mode).

I was able to enable VFs on both ports. First Port 0:

Edit /etc/cxl0.conf:

```
PF {device:"cxl0";num_vfs:2;}
DEFAULT {passthrough:true;}
```

and in /etc/rc.conf:

```
iovctl_files="/etc/cxl0.conf"
```

Reboot and pciconf lists the VFs as PCI pass-through devices:

```
ppt0@pci0:3:0:8:       class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt1@pci0:3:0:12:       class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```

where 3:0:8 and 3:0:12 correspond to VFs on Port 0 (3:0:0). Note that there's no change to the pciconf output from above (for t5iov[0-3]).

Now for Port 1:

Edit /etc/cxl1.conf:

```
PF {device:"cxl1";num_vfs:2;}
DEFAULT {passthrough:true;}
```

and in /etc/rc.conf:

```
iovctl_files="/etc/cxl0.conf /etc/cxl1.conf"
```

Reboot and there will now be four ppt devices:

```
ppt0@pci0:3:0:8:    class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt1@pci0:3:0:12:    class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt2@pci0:3:0:9:    class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
ppt3@pci0:3:0:13:    class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```

where 3:0:9 and 3:0:13 correspond to VFs on Port 1 (3:0:1).


----------



## Phishfry (Aug 18, 2019)

That is great news. I moved to Intel cards with 10G for a spell for testing. Mainly x520 Silicom and Advantech variants.
I just got my 4 quad Chelsio T540 for a 10G soft switch project and ordered a bunch more!

What have you found for the maximum VF on the T5xx cards? Same as T4xx?
These T5xx series seem to run cooler, but still require active cooling.
Thoughts? Have you monitored your temps with the sysctl? What do you use for cooling?
I have a 120mm PWM fan set to full blast@2700rpm over 3 cards installed and I might need to step it up.
That is with near zero load.
How much do the temps rise in production setting is my question.

The more I dig into the documentation on these Chelsio cards the more I like.
The firewall features alone are superb.
Not hard to load rules either with cxgbetool in base. You can even flash the firmware from it.


----------



## openjohn (Aug 24, 2019)

I've hit a couple of downstream snags. I'm still tinkering.

First is that my test machine doesn't support VT-d, which is required for PCI passthru to VMs. So the ppt devices can be configured in the host, but Bhyve complains:

```
# tail /vm/vm1/vm-bhyve.log
fatal; pci passthrough not supported on this system (no VT-d or amdvi)
```

I decided to try using the VFs in the host (non-passthrough), passing them to jails. I made some progress with this. In /boot/loader.conf

```
if_cxgbe_load="YES"
if_cxgbev_load="YES"
```

and pciconf shows:

```
t5vf0@pci0:3:0:8:       class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
t5vf1@pci0:3:0:12:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
t5vf2@pci0:3:0:16:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
t5vf3@pci0:3:0:20:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
none5@pci0:3:0:9:       class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
none6@pci0:3:0:13:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
none7@pci0:3:0:17:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
none8@pci0:3:0:21:      class=0x020000 card=0x00001425 chip=0x58011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller [VF]'
    class      = network
    subclass   = ethernet
```

and the reason we have none5-8 is because in dmesg:

```
t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
t5vf1: <Chelsio T520-CR VF> at device 0.12 on pci3
cxlv1: <port 0> on t5vf1
cxlv1: Ethernet address: 06:44:2b:b8:90:01
cxlv1: 2 txq, 1 rxq (NIC)
t5vf1: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
t5vf2: <Chelsio T520-CR VF> at device 0.16 on pci3
cxlv2: <port 0> on t5vf2
cxlv2: Ethernet address: 06:44:2b:b8:90:02
cxlv2: 2 txq, 1 rxq (NIC)
t5vf2: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
t5vf3: <Chelsio T520-CR VF> at device 0.20 on pci3
cxlv3: <port 0> on t5vf3
cxlv3: Ethernet address: 06:44:2b:b8:90:03
cxlv3: 2 txq, 1 rxq (NIC)
t5vf3: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
t5vf4: <Chelsio T520-CR VF> at device 0.9 on pci3
t5vf4: FW reset failed: 1
device_attach: t5vf4 attach returned 1
t5vf4: <Chelsio T520-CR VF> at device 0.13 on pci3
t5vf4: FW reset failed: 1
device_attach: t5vf4 attach returned 1
t5vf4: <Chelsio T520-CR VF> at device 0.17 on pci3
t5vf4: FW reset failed: 1
device_attach: t5vf4 attach returned 1
t5vf4: <Chelsio T520-CR VF> at device 0.21 on pci3
t5vf4: FW reset failed: 1
device_attach: t5vf4 attach returned 1
```

I traced the error message into the driver code:

```
/*
     * Some environments do not properly handle PCIE FLRs -- e.g. in Linux
     * 2.6.31 and later we can't call pci_reset_function() in order to
     * issue an FLR because of a self- deadlock on the device semaphore.
     * Meanwhile, the OS infrastructure doesn't issue FLRs in all the
     * cases where they're needed -- for instance, some versions of KVM
     * fail to reset "Assigned Devices" when the VM reboots.  Therefore we
     * use the firmware based reset in order to reset any per function
     * state.
     */
    rc = -t4vf_fw_reset(sc);
    if (rc != 0) {
        device_printf(dev, "FW reset failed: %d\n", rc);
        goto done;
    }
```

So there's a firmware reset happening, which is failing for the VFs associated with Port 1.

Nonetheless, the VFs that don't have errors in dmesg appear as interfaces in ifconfig:

```
cxlv0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether XX:XX:XX:XX:XX:XX
    media: Ethernet 10Gbase-SR <full-duplex> (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxlv1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether XX:XX:XX:XX:XX:XX
    media: Ethernet 10Gbase-SR <full-duplex> (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxlv2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether XX:XX:XX:XX:XX:XX
    media: Ethernet 10Gbase-SR <full-duplex> (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
cxlv3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP>
    ether XX:XX:XX:XX:XX:XX
    media: Ethernet 10Gbase-SR <full-duplex> (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```


----------



## Nyakov (Dec 10, 2020)

Any update on T520-CR/T540-CR passthrough ?

I cannot passthrough VFs no mater what I do.
I get this error:

```
bhyve: PCI device at 1/1/3 is not using the ppt(4) driverdevice emulation initialization error: No such file or directory
```


----------

