# strange behavior of setfib since FreeBSD 12.0



## icecoke (Dec 11, 2019)

Hi everyone!

We are using jails in FreeBSD since years (9.x and up) without problems. Since then we have two network cards in the (xen virtualized) machines, where one is private (10.0.0.0/8) and one is public (here changed to 1.2.3.0/20). To have the routing done, we use setfib for the public network, which worked always without problems.

Since a few month, we move a few hundred machines to 12.0-RELEASE-p10 and now (or maybe earlier, since we do not check each machine, each day or so) we experience that the sendmail on port 25 sometimes stop to talk correctly with the outside world. The communication from and to machines within the 1.2.3.0/20 keeps working. If we restart the sendmail process, then all communication in all networks is running again (by now, we can't say when and how fast the error will come back).

In the host, there is a sendmail on port 25 on loopback, not on any of the NICs.

Currently we're just digging around and one clear difference at least between 10.x and 12.0 seems to be the view of netstat -nr with setfib from the host view:

in 10.x:

[root@10-0-4-144-host ~]# netstat -rn

```
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            10.0.0.1           UGS         xn0
10.0.0.0/16        link#5             U           xn0
10.0.4.144         link#5             UHS         lo0
127.0.0.1          link#4             UH          lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#4                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%lo0/32                     ::1                           U           lo0
```

[root@10-0-4-142-host ~]# setfib 2 netstat -rn


```
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags      Netif Expire
default            1.2.3.4            UGS         xn1
1.2.3.0/20         link#6             U           xn1
1.2.3.8            link#6             UHS         lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
fe80::%lo0/64                     link#4                        U           lo0
```

In 12.0:

[root@10-0-4-144-host ~]# netstat -rn

```
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.0.1           UGS         xn0
10.0.0.0/16        link#4             U           xn0
10.0.4.144         link#4             UHS         lo0
1.2.3.0/20         link#5             U           xn1
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
```

[root@10-0-4-144-host ~]# setfib 2 netstat -rn

```
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            1.2.3.4            UGS         xn1
10.0.0.0/16        link#4             U           xn0
1.2.3.0/20         link#5             U           xn1
1.2.3.8            link#5             UHS         lo0
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
ff02::/16                         ::1                           UGRS        lo0
```


/etc/rc.conf:


```
ifconfig_xn0="inet 10.0.4.144/16 mtu 1500"
ifconfig_xn1="inet 1.2.3.8/20 fib 2"

defaultrouter="10.0.0.1"
static_routes="vsnetwork vsdefault"
route_vsnetwork="-net 1.2.3.0/20 -interface xn1 -fib 2"
route_vsdefault="default 1.2.3.4 -fib 2"
```

As one can see, on FreeBSD 10.x the netstat don't show up the routing of the other fibs as expected. The 12.0 shows it, what is not expected.

So my question is: is this just a 'false' printing in the FreeBSD 12.0, or is it wrong and therefore any change between 10.x and 12.0 regarding setfib? So is there a new way to set this up? Or is it a bug?

Many thanks in advance for your input!

Jimmy


----------



## icecoke (Dec 16, 2019)

It seems to be a bug in FreeBSD > 10 or at least in FreeBSD 12.0, as these route, which only were created in 12.x (which never set explicit) needs to be removed to have everything working (from the example nets above):


```
route delete -net 1.2.3.0/20 -iface xn1 -fib 0
route delete 127.0.0.1 -iface lo0 -fib 2
route delete -net 10.0.0.0/16 -iface lo0 -fib 2
```

Sorry noone had input here within the forum, but maybe this will help others out who is working with jails, `setfib` and different networks/routes in FreeBSD >= 12.0


----------



## amiramix (Apr 19, 2020)

Could that be related to not having `net.add_addr_allfibs=0` in /etc/sysctl.conf? Especially that it looks like up to FreeBSD 10 it could be placed in /etc/rc.conf, see a discussion on the stable list group.


----------



## icecoke (Apr 19, 2020)

Thanks! Nice finding! That solution is working for me, too. We never had this in our FreeBSD 10.x machines, so we run into this as it seems `net.add_addr_allfibs=1` seems to be the default (for whatever reason...).


----------

