# FreeBSD 8.3 / 10.0 and Intel 10GbE Virtual Functions



## skpio (Aug 5, 2014)

I've run into a one-way transmission issue with FreeBSD and Virtual Functions on an Intel 10GbE X520. So far we've tested:


Ubuntu 12.04 LTS
CentOS 6.5
pfSense 2.1 / FreeBSD 8.3
pfSense 2.2 / FreeBSD 10.0 (alpha)
FreeBSD 8.3-STABLE

All are instances are running on top of KVM installed on Ubuntu 14, hosted on Intel Grizzly Pass OEM hardware with the above-mentioned card installed. The card's two 10GbE ports are connected back-to-back.

In our test case, all instances above have a VF inside VLAN 100 and are bridged via the 82599 controller's internal L2 'switch.' The Ubuntu and CentOS instances can reach each other on their VF/VLAN100 interfaces, but any FreeBSD instance spun up cannot. `tcpdump` on a CentOS instance and a FreeBSD instance show the FreeBSD instance send an ARP request, the CentOS instance receive it and respond, but it never reaches the FreeBSD instance. Since this is not an issue on Ubuntu or CentOS, and has persisted across two versions of FreeBSD and two Intel VF driver versions, I have to assume it's unique to FreeBSD.

`freebsd1# tcpdump -i ix0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes
21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel`

`[root@centos1 ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28
17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28
^C
24 packets captured
24 packets received by filter
0 packets dropped by kernel`

`freebsd1# uname -a
FreeBSD freebsd1 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Mon Apr  9 21:23:18 UTC 2012     [email=root@mason.cse.buffalo.edu]root@mason.cse.buffalo.edu[/email]:/usr/obj/usr/src/sys/GENERIC  amd64`

`freebsd1# dmesg | grep Virtual
CPU: QEMU Virtual CPU version 2.0.0 (2793.29-MHz K8-class CPU)
ix0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.1.2> mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0`

`freebsd1# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 52:54:00:44:99:6c
        inet6 fe80::5054:ff:fe44:996c%ix0 prefixlen 64 scopeid 0x3
        inet 192.168.100.253 netmask 0xffffff00 broadcast 192.168.100.255
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect
        status: active`

Note that pfSense runs v1.1.4 of the Intel VF driver. Intel only appears to offer a VF driver for the e1000 type NIC for FreeBSD. They do have 'standard' drivers for 10GbE cards for FreeBSD, however. Our issue is identical in description to this post.

However, there is no "Simulated MSI Support" option in our BIOS and I believe that particular option was unique to his board based on Intel's BIOS release notes. Another individual ran into a similar problem with FreeBSD 10.0.

I'm currently at a loss. I can provide whatever additional information may be helpful; I'm not sure what else is useful at this point. I've dumped some additional info below.

On the host, SR-IOV and IOMMU are enabled. The 10GbE card is on PCI bus 81:00:

`root@ubuntu:~# dmesg | grep Intel | grep ixgbe
[    8.690398] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k
[    8.690398] ixgbe: Copyright (c) 1999-2013 Intel Corporation.
[    8.950982] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    9.211614] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection`

VFs are enabled at the host level:

`root@ubuntu:~# lspci -s 81:
81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)`

The virtual functions with several instances spun up. VF3 is centos1, VF6 is freebsd1.

`root@ubuntu:~# ip link show dev eth4
5: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 3 MAC 52:54:00:db:57:b2, vlan 100, spoof checking on, link-state auto
    vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 6 MAC 52:54:00:44:99:6c, vlan 100, spoof checking on, link-state auto
    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto`

KVM is also pleased:

`root@ubuntu:~# virsh nodedev-list | grep 81
pci_0000_81_00_0
pci_0000_81_00_1
pci_0000_81_10_0
pci_0000_81_10_1
pci_0000_81_10_2
pci_0000_81_10_3
pci_0000_81_10_4
pci_0000_81_10_5
pci_0000_81_10_6
pci_0000_81_10_7
pci_0000_81_11_0
pci_0000_81_11_1
pci_0000_81_11_2
pci_0000_81_11_3
pci_0000_81_11_4
pci_0000_81_11_5
pci_0000_81_11_6
pci_0000_81_11_7
pci_0000_81_12_0
pci_0000_81_12_1
pci_0000_81_12_2
pci_0000_81_12_3
pci_0000_81_12_4
pci_0000_81_12_5
pci_0000_81_12_6
pci_0000_81_12_7
pci_0000_81_13_0
pci_0000_81_13_1
pci_0000_81_13_2
pci_0000_81_13_3
pci_0000_81_13_4
pci_0000_81_13_5
pci_0000_81_13_6
pci_0000_81_13_7`

The FreeBSD instance in question is on pci_0000_81_11_4, which maps to VF6 above. 

KVM XML for the freebsd1's interface:


```
<interface type='hostdev' managed='yes'>
      <mac address='52:54:00:44:99:6c'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x81' slot='0x11' function='0x4'/>
      </source>
      <vlan>
        <tag id='100'/>
      </vlan>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
```

Thanks to anyone who reads and responds. I could really use some help.

Thanks!

*EDIT:* gonzopancho on reddit directed me to this link, which has a number of patches written by Ryan Stone. It seems like this may be a known issue. At this point we're just looking for confirmation that we aren't crazy. I've felt like this for a week.


----------

