# VNET Jails and SR-IOV virtual functions



## UrsusDominatus (Dec 14, 2020)

Hi all. I have a routing problem with my VNET jails setup and I need help.
Some time ago, I setup vnet jails infrastructure using epair+bridge interfaces and everything works flawlessly. Recently I learned SR-IOV feature, read SR-IOV on FreeBSD paper, and decide to try it myself.
I've managed to create virtual devices for Chelsio T420-CR NIC, and successfully assign them to my jails. At first I think everything works as usual, but soon one issue is appear. I cannot reach physical/virtual interfaces IP from other physical/virtual interfaces *on same NIC*






On above diagram I tried to describe current configuration. Grayed VF captions indicates virtual interfaces that disappear from host and assigned to corresponding jails.
I can reach gateway from any of them without problems:

```
root@jail-host:/ # ping -S 10.10.30.2 -c 2 10.10.30.1
PING 10.10.30.1 (10.10.30.1) from 10.10.30.2: 56 data bytes
64 bytes from 10.10.30.1: icmp_seq=0 ttl=64 time=0.225 ms
64 bytes from 10.10.30.1: icmp_seq=1 ttl=64 time=0.074 ms
...

root@jail1:/ # ping -S 10.10.30.11 -c 2 10.10.30.1
PING 10.10.30.1 (10.10.30.1) from 10.10.30.11: 56 data bytes
64 bytes from 10.10.30.1: icmp_seq=0 ttl=64 time=0.079 ms
64 bytes from 10.10.30.1: icmp_seq=1 ttl=64 time=0.087 ms
...

root@jail0:/ # ping -S 10.10.30.10 -c 2 10.10.30.1
PING 10.10.30.1 (10.10.30.1) from 10.10.30.10: 56 data bytes
64 bytes from 10.10.30.1: icmp_seq=0 ttl=64 time=0.072 ms
64 bytes from 10.10.30.1: icmp_seq=1 ttl=64 time=0.074 ms
```

But I cannot reach any of target in any of following combinations: PF->VF, VF->PF, VF->VF

```
root@jail-host:~ # ping -S 10.10.30.2 -c 2 10.10.30.11
PING 10.10.30.11 (10.10.30.11) from 10.10.30.2: 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
^C

root@jail0:/ # ping -S 10.10.30.10 -c 2 10.10.30.2
PING 10.10.30.2 (10.10.30.2) from 10.10.30.10: 56 data bytes

--- 10.10.30.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

root@jail0:/ # ping -S 10.10.30.10 -c 2 10.10.30.11
PING 10.10.30.11 (10.10.30.11) from 10.10.30.10: 56 data bytes

--- 10.10.30.11 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
```

What have I tried:
- Experiment with routes. No success.
- Adding "gateway_enable="YES"". No success.
- Reading about virtual switch function in Chelsio and Intel NICs (mentioned here and here). But no actual steps were taken because no configuration options were found.

I think I miss something obivious but fundamental. Thank you for your help in advance.


----------



## driesm (Jan 17, 2021)

Could you bring this up on the mailing list? I'm interested in this topic as well. I think this is by design, that you create a direct access path to the physical NIC by giving it a virtual function.


----------

