# IPv6 routing for VNET jails on FreeBSD 13-RELEASE



## codetrotter (Oct 6, 2021)

Hi,

I have a IPv6 address block of prefix size 64. On my host computer running FreeBSD 13-RELEASE I have three jails:

`jls`


```
JID  IP Address      Hostname                      Path
     1                  svcfw                         /usr/jail/svcfw
     2                  www                           /usr/jail/www
     3                  ifee                          /usr/jail/ifee
```

Host network adapter connected to global Internet:

`ifconfig vtnet0`


```
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4804bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,LINKSTATE,TXCSUM_IPV6>
    ether 56:00:03:7a:2c:a1
    inet 192.248.184.48 netmask 0xfffffe00 broadcast 192.248.185.255
    inet6 fe80::5400:3ff:fe7a:2ca1%vtnet0 prefixlen 64 scopeid 0x1
    inet6 2001:19f0:6c01:2033::2 prefixlen 65
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

Contents of my /etc/rc.conf:


```
hostname="login"

clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
static_routes="linklocal"

route_linklocal="-net 169.254.0.0/16 -interface vtnet0"
ifconfig_vtnet0="inet 192.248.184.48 netmask 255.255.254.0"
defaultrouter="192.248.184.1"
ifconfig_vtnet0_ipv6="inet6 2001:19f0:6c01:2033::2 prefixlen 65"
ipv6_default_interface="vtnet0"
# XXX:
#  - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233283#c4
#  - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233283#c16
ipv6_defaultrouter="fe80::fc00:3ff:fe7a:2ca1%vtnet0"
cloned_interfaces="bridge0 epair0"
ifconfig_epair0a_ipv6="inet6 2001:19f0:6c01:2033:8000::1 prefixlen 65"
ifconfig_epair0a="inet 10.22.0.1 netmask 255.255.0.0"

# Private network: net5ab50b04a6b66
ifconfig_vtnet1="inet 10.9.96.5 netmask 255.255.240.0 mtu 1450"

#pf_enable="YES"
#pf_rules="/etc/pf.conf"
#pflog_enable="YES"
#pflog_logfile="/var/log/pflog"

sshd_enable="YES"

dumpdev="NO"
zfs_enable="YES"
devmatch_blacklist="virtio_random.ko"
gateway_enable="YES"
ipv6_gateway_enable="YES"
wireguard_enable="YES"
wireguard_interfaces="wg0"
jail_enable="YES"
```

Networking from host is working as it should:

`fetch -6 -o /dev/null http://example.com`


```
/dev/null                                             1256  B   10 MBps    00s
```

`ping6 -c 4 google.com`


```
PING6(56=40+8+8 bytes) 2001:19f0:6c01:2033::2 --> 2a00:1450:4001:802::200e
16 bytes from 2a00:1450:4001:802::200e, icmp_seq=0 hlim=119 time=0.832 ms
16 bytes from 2a00:1450:4001:802::200e, icmp_seq=1 hlim=119 time=0.793 ms
16 bytes from 2a00:1450:4001:802::200e, icmp_seq=2 hlim=119 time=0.867 ms
16 bytes from 2a00:1450:4001:802::200e, icmp_seq=3 hlim=119 time=0.827 ms
--- google.com ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.793/0.830/0.867/0.026 ms
```

Main network adapter of the first jail:

`doas jexec svcfw ifconfig bridge0`


```
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:e2:24
    inet6 2001:19f0:6c01:2033:8000::2 prefixlen 65
    inet 10.22.0.2 netmask 0xffff0000 broadcast 10.22.255.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000
    member: epair0b flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 2000
    groups: bridge
    nd6 options=8001<PERFORMNUD,DEFAULTIF>
```

Ping jail from host works as it should:

`ping6 -c 4 2001:19f0:6c01:2033:8000::2`


```
PING6(56=40+8+8 bytes) 2001:19f0:6c01:2033:8000::1 --> 2001:19f0:6c01:2033:8000::2
16 bytes from 2001:19f0:6c01:2033:8000::2, icmp_seq=0 hlim=64 time=0.077 ms
16 bytes from 2001:19f0:6c01:2033:8000::2, icmp_seq=1 hlim=64 time=0.062 ms
16 bytes from 2001:19f0:6c01:2033:8000::2, icmp_seq=2 hlim=64 time=0.080 ms
16 bytes from 2001:19f0:6c01:2033:8000::2, icmp_seq=3 hlim=64 time=0.062 ms
--- 2001:19f0:6c01:2033:8000::2 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.062/0.070/0.080/0.008 ms
```

Ping host from jail works too as it should:

`doas jexec svcfw ping6 -c 4 2001:19f0:6c01:2033:8000::1`


```
PING6(56=40+8+8 bytes) 2001:19f0:6c01:2033:8000::2 --> 2001:19f0:6c01:2033:8000::1
16 bytes from 2001:19f0:6c01:2033:8000::1, icmp_seq=0 hlim=64 time=0.068 ms
16 bytes from 2001:19f0:6c01:2033:8000::1, icmp_seq=1 hlim=64 time=0.064 ms
16 bytes from 2001:19f0:6c01:2033:8000::1, icmp_seq=2 hlim=64 time=0.059 ms
16 bytes from 2001:19f0:6c01:2033:8000::1, icmp_seq=3 hlim=64 time=0.060 ms
--- 2001:19f0:6c01:2033:8000::1 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.059/0.063/0.068/0.004 ms
```

Configuration of host-side pair connected to bridge:

`ifconfig epair0a`


```
epair0a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:a8:b4:85:57:0a
    inet 10.22.0.1 netmask 0xffff0000 broadcast 10.22.255.255
    inet6 fe80::a8:b4ff:fe85:570a%epair0a prefixlen 64 scopeid 0x5
    inet6 2001:19f0:6c01:2033:8000::1 prefixlen 65
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

Ordinarily, pf is doing NAT for IPv4. Currently I have disabled pf so that pf is not running while I am trying to get global IPv6 to work for the jails.

The host is configured to route IPv6 traffic, cf /etc/rc.conf that I included earlier in this message.

Here is what the jail says about the route to an IPv6 address on the internet:

`doas jexec svcfw route -6 get 2a00:1450:400f:802::200e`


```
route to: 2a00:1450:400f:802::200e
destination: default
       mask: default
    gateway: 2001:19f0:6c01:2033:8000::1
        fib: 0
  interface: bridge0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
```

Should be sent to bridge0, looks good to me.

Here is what the host tells me about the routes for google.com and for the jail respectively:

`route -6 get google.com`


```
route to: fra24s01-in-x0e.1e100.net
destination: default
       mask: default
    gateway: fe80::fc00:3ff:fe7a:2ca1%vtnet0
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
```

`route -6 get 2001:19f0:6c01:2033::2`


```
route to: 2001:19f0:6c01:2033::2
destination: 2001:19f0:6c01:2033::2
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0
```

Looks mostly good. Dunno why the latter route says the loopback interface lo0 instead of the epair interface epair0a. Is this a problem? Or is something else causing the problem?

The problem being that I cannot reach the public Internet by IPv6 from my jail, even though IPv6 routing on the host is on and the host firewall (pf) is off, and the host can speak with the Internet over IPv6 just fine and I am using addresses from my assigned /64. (Assigning more addresses from the /64 to the host works fine btw.)

When I try to even ping something on the Internet by IPv6 that worked fine from the host, doesn't work from the jail:

`doas jexec svcfw ping6 -c 4 2a00:1450:400f:802::200e`


```
PING6(56=40+8+8 bytes) 2001:19f0:6c01:2033:8000::2 --> 2a00:1450:400f:802::200e
--- 2a00:1450:400f:802::200e ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
```

Doesn't work.

What am I missing? What am I doing wrong? Been going on at this for hours.


----------



## SirDice (Oct 6, 2021)

The smallest prefix you can use for a subnet is a /64. You really can't split this up any further as you're used to with IPv4.


----------

