# Quagga fails to set route on FreeBSD 13.0-Release



## von_Gaden (Apr 27, 2021)

Hi all,
I found an unexpected problem running net/quagga after upgrade FreeBSD 12.2-RELEASE to FreeBSD 13.0-RELEASE.
`bgpd` seems to work fine, but obviously `zebra` is unable to set the default route in the Kernel. In zebra.log I have:

```
ZEBRA: Try to set TE Link Param
ZEBRA: Try to set TE Link Param
ZEBRA: if_ioctl(SIOCGIFMEDIA) failed: Invalid argument
ZEBRA: rtm_write: write : Invalid argument (22)
ZEBRA: kernel_rtm_ipv4: 0.0.0.0/0: rtm_write() unexpectedly returned -5 for command RTM_ADD
ZEBRA: Try to set TE Link Param
ZEBRA: Try to set TE Link Param
ZEBRA: rtm_write: write : Invalid argument (22)
ZEBRA: kernel_rtm_ipv4: 0.0.0.0/0: rtm_write() unexpectedly returned -5 for command RTM_ADD
ZEBRA: rtm_write: write : Invalid argument (22)
ZEBRA: kernel_rtm_ipv4: 0.0.0.0/0: rtm_write() unexpectedly returned -5 for command RTM_ADD
```
net/quagga was built in my ports-mgmt/poudriere with the default options as for FreeBSD 12.2. I've tested with the package from the official FreeBSD repository with the same result.
Some details for my setup:
I have a stub IPv4 network, part of a bigger AS. BGP is used to switch between three links - main and two backups. We have to announce our subnet to the upstream and get default route.
Is this related to the lack of maintainer to the net/quagga port?

Thank you in advance.


----------



## SirDice (Apr 27, 2021)

Do you use FIBs? This might be a reason for the failure:

```
20201108:
        Default value of net.add_addr_allfibs has been changed to 0.
        If you have multi-fib configuration and rely on existence of all
        interface routes in every fib, you need to set the above sysctl to 1.
```


----------



## von_Gaden (Apr 27, 2021)

No, I have just one routing table with several static routes inside the LAN and one default route acquired via BGP.


----------



## covacat (Apr 27, 2021)

try to ktrace/truss, get more info on the failed syscalls


----------



## von_Gaden (May 3, 2021)

I'll do my best but it will take some time. It's production system and I have to fit in a maintenance window.


----------



## Donald Baud (May 5, 2021)

You might want to try net/frr7
It is the descendant of zebra -> quagga -> frr
from the website  frrrouting.org
"
FRR has its roots in the Quagga project. In fact, it was started by many long-time Quagga developers who combined their efforts to improve on Quagga’s well-established foundation in order to create the best routing protocol stack available.
"


----------



## sko (May 5, 2021)

Donald Baud said:


> You might want to try net/frr7
> [...]



If changing the BGP daemon is on the table (and with the quagga port being orphaned I would highly suggest looking for other options!), I'd suggest looking at net/openbgpd
I'm running our whole multi-homed, multi-site routing infrastructure with OpenBGPd (on Open- and FreeBSD) for internal and external routing, and couldn't be more pleased with it. I especially like the clear, simple and easy to read config (similar syntax to PF) which prevents a lot of headaches in the first place. It also nicely integrates with multiple FIBs (or routing domains on OpenBSD) and for very special cases (or testing) you can even run multiple instances of it (in different routing domains).

It also comes with bgpctl for manual interaction/configuration/management and which is great for scripting and (ab)using BGP for other purposes (e.g. distributing blacklists between nameservers, firewalls and mailservers with some scripting-ducttape around bgpctl and pfctl/spamdb)


----------



## von_Gaden (May 17, 2021)

Thank you very much for the helpful replies!
I'll test both routing daemons later.
Sadly now I have troubles with unexpected kernel panics in FreeBSD 13.0-Release. I believe they are related to `ipfw` and `natd` and are discussed here or in PRs. Without official patches (with unclear schedule yet) it's pointless. I've already managed to migrate several production systems to 13.0-Release and I consider reinstalling them with 12.2-Release.


----------



## cmilos (Jul 16, 2021)

sko said:


> If changing the BGP daemon is on the table (and with the quagga port being orphaned I would highly suggest looking for other options!), I'd suggest looking at net/openbgpd
> I'm running our whole multi-homed, multi-site routing infrastructure with OpenBGPd (on Open- and FreeBSD) for internal and external routing, and couldn't be more pleased with it. I especially like the clear, simple and easy to read config (similar syntax to PF) which prevents a lot of headaches in the first place. It also nicely integrates with multiple FIBs (or routing domains on OpenBSD) and for very special cases (or testing) you can even run multiple instances of it (in different routing domains).
> 
> It also comes with bgpctl for manual interaction/configuration/management and which is great for scripting and (ab)using BGP for other purposes (e.g. distributing blacklists between nameservers, firewalls and mailservers with some scripting-ducttape around bgpctl and pfctl/spamdb)


I had the same issue as von_Gaden installing Quagga on a fresh 13.0-RELEASE install. Luckily I have a lot of experience with both OpenBGPD and FRR. I've used OpenBGPD on FreeBSD for years and it's rock solid and well featured enough for my purposes.
I use FRR extensively for our routing at work, we have it on 3 "platforms" - Cumulus Linux, FreeBSD 12/13 and pfSense (FreeBSD). pfSense have actually deprecated Quagga, I used it on pfSense in the older versions.
If you're only using it for BGP then OpenBGPD should do what you need, if you want to use OSPF (we do) then go for FRR.


----------



## von_Gaden (Jul 22, 2021)

Today I was able to test net/openbgpd6, but with no success. I've got the following messages:

```
bgpd[2301]: PF_KEY not available, disabling ipsec
bgpd[2303]: neighbor 19.14.25.25 (bak2-1G): Received multi protocol capability:  unknown AFI 1, safi 2 pair
bgpd[2303]: neighbor 19.14.23.21 (main): Received multi protocol capability:  unknown AFI 1, safi 2 pair
bgpd[2303]: peer closed imsg connection
bgpd[2302]: peer closed imsg connection
bgpd[2303]: SE: Lost connection to parent
bgpd[2304]: peer closed imsg connection
bgpd[2302]: fatal in RDE: Lost connection to parent
bgpd[2303]: neighbor 19.14.23.21 (main): sending notification: Cease, administratively down
bgpd[2304]: fatal in RTR: Lost connection to parent
bgpd[2303]: neighbor 19.14.23.25 (bak1-10G): sending notification: Cease, administratively down
bgpd[2303]: neighbor 19.14.25.25 (bak2-1G): sending notification: Cease, administratively down
```
I also couldn't figure out how to use weight for neighbours because the 10G connections should be always preferred over the last backup of 1 Gbit.
I guess I should try net/frr7


----------



## sko (Jul 22, 2021)

I'm not sure about those error messages, but is this by any chance some kind of MPLS VPN setup?



von_Gaden said:


> I also couldn't figure out how to use weight for neighbours because the 10G connections should be always preferred over the last backup of 1 Gbit.



outgoing metric can be set this way (peering with AS 65012 is our main uplink, AS 65001 our backup):

```
# set route preferences
match to AS 65001 set { community 65000:10100, metric 100 }
match to AS 65012 set { community 65000:10200, metric 0 }
```

Internally I personally just use a fixed set of localprefs for different communities and apply those to incoming routes:

```
match from AS 65001 set { community 65000:10100 }
match from AS 65012 set { community 65000:10200 }

[...]

match from any community *:10050 set { localpref 50 }
match from any community *:10100 set { localpref 100 }
match from any community *:10150 set { localpref 150 }
match from any community *:10200 set { localpref 200 }
```

I apply those communities 10050-10200) at the edge routers and all internal gateways/routers just need that last block of community to localpref translation. This way I can quickly change route preferences at the edge routers by just changing the community string (e.g. in case of routing problems with the primary uplink or when another uplink is added)

You could also match to groups or peers instead of AS; the syntax is quite flexible and very similar to the one used in pf.conf.


FTR: I use OpenBGPD almost exclusively on OpenBSD and only with very basic rulesets on very few FreeBSD hosts, so I'm not that familiar with possible differences between the 'original' and FreeBSD versions.


----------



## von_Gaden (Jul 24, 2021)

I don't have MPLS VPN. The setup is very simple - my network is "stub", the uplinks belong to the same AS and they announce the default route only. The first one is main and we have two fallback but only one link works at a given time. Here is my net/quagga bgpd.conf:


```
! Note: the IPs and AS numbers are bogus
!
hostname mygate
log file /var/log/quagga/bgpd.log debugging
!
router bgp 65001
 bgp router-id 19.14.15.1
 bgp log-neighbor-changes
 network 19.14.15.0/24
 neighbor 19.14.23.21 remote-as 65001
 neighbor 19.14.23.21 weight 100
!
 neighbor 19.14.23.25 remote-as 65001
 neighbor 19.14.23.25 weight 100
!
 neighbor 19.14.25.25 remote-as 65001
 neighbor 19.14.25.25 weight 90
!
 address-family ipv6
 exit-address-family
 exit
```

I suppose I *should* have tested with net/openbgpd instead...


----------

