# Default route not surviving reboot and randomly disappearing



## LITMK (Jul 10, 2020)

Hello,

I'm running FreeBSD 12.1-RELEASE-p3 and my /etc/rc.conf has my network set up like this:


```
ifconfig_igb1="inet 10.0.0.20 netmask 255.255.255.0"
defaultrouter="10.0.0.1"
```

Lately when I reboot the machine the default route is not set and I have to add it manually with: 


```
route add default 10.0.0.1
```

Sometimes after being up for a while the default route disappears and I've confirmed this with netstat. I then have to add it manually again. 

I don't know where to start diagnosing this and search engine results have turned up a bug from FreeBSD 9.0 where the default route would change to another address under high load, but that's not my problem. 

I appreciate any help or suggestions on where to look or what to check.


----------



## SirDice (Jul 10, 2020)

LITMK said:


> Sometimes after being up for a while the default route disappears and I've confirmed this with netstat.


This could be the result of the interface disappearing and reappearing. When an interface is removed (for whatever reason) the accompanying routes also disappear. When the interface comes back those routes will still be gone. Check your /var/log/messages, does that provide any clues to the reason?


----------



## LITMK (Jul 10, 2020)

You figured it out. Thank you!

The logs show the interface going down and coming up a few times. 

I don't run a huge amount of traffic on this machine. About a dozen jails and a  bhyve Ubuntu game server.

Do you have any ideas as to what could be causing this problem? I really have no idea where to start looking.

It's onboard NIC,  Intel server board (Intel S5520HC).


----------



## SirDice (Jul 10, 2020)

LITMK said:


> The logs show the interface going down and coming up a few times.
> ...
> Do you have any ideas as to what could be causing this problem?


There could be several reasons.  The first could be that the card is faulty and it's just limping along until it completely dies. Another reason  could be a bad cable causing the connection to to be down intermittently. It could also be  the port  of the switch it  is connected to.  What exactly is being reported in /var/log/messages?


----------



## Lamia (Jul 10, 2020)

SirDice  has said it all. If you however want to restore you route you can enter it into rc.local, where the routes are immediately loaded on start-up AND you could use Cron to periodically run the file - 

```
0/10 * * * * sh /etc/rc.local >/dev/null 2>=1
```


----------



## mickey (Jul 10, 2020)

LITMK said:


> I'm running FreeBSD 12.1-RELEASE-p3 ...


I'm not sure if this is related to the problem you are experiencing, but there has been an ERRATA notice regarding a problem with the igb driver, so updating your system might be something worth considering.


----------



## Lamia (Jul 10, 2020)

mickey said:


> I'm not sure if this is related to the problem you are experiencing, but there has been an ERRATA notice regarding a problem with the igb driver, so updating your system might be something worth considering.


I saw the Igb interface act up recently - altq queueing broke invariably pf. A kernel and world upgrade fixed it. We're not informed of his network interfaces and hardware.


----------



## mickey (Jul 10, 2020)

Lamia said:


> I saw the Igb interface act up recently - altq queueing broke invariably pf. A kernel and world upgrade fixed it. We're not informed of his network interfaces and hardware.


I'd say we are:


LITMK said:


> ```
> ifconfig_igb1="inet 10.0.0.20 netmask 255.255.255.0"
> ```


----------



## SirDice (Jul 10, 2020)

With 12.0 the igb driver was integrated into the em(4) driver. It used to be a separate driver. This integration effort may have introduced bugs. I have four igb  interfaces on a SuperMicro server which is running 12-STABLE since it became available. So far I haven't encountered any serious issues with it. This doesn't rule out bugs though, I may have been lucky not to have been bitten by them.

I would start by  replacing the cable. If possible also try a different switch port. Those are easy to try without having to take the whole thing apart.


----------



## VladiBG (Jul 10, 2020)

It doesn't matter if there's cable or connection to the interface when he's using static route and static ip address configuration. The routing changes are only visible on the console so he can't see them in the /var/log/messages

Most probable cause is that the driver of the interface is loaded after the route fail or he already have another default route in the table.

After the reboot check the console output for:


```
route: writing to routing socket: Network is unreachable
add net  default: gateway 10.0.0.1 fib 0: Network is unreachable
```

If there's no connection to the interface (aka disconnected cable) but the driver is loaded before the route it will get the route with interface down.

Edit:
If you have another default route on igb0 interface configured by DHCP will cause the same issue. As you can't have 2 default gw
you will see the error
`add net default: gateway 10.0.0.1 fib 0: route already in table`


----------



## SirDice (Jul 10, 2020)

VladiBG said:


> The routing changes are only visible on the console so he can't see them in the /var/log/messages


In that  case  I would recommend enabling this  in /etc/syslog.conf:

```
# uncomment this to log all writes to /dev/console to /var/log/console.log
# touch /var/log/console.log and chmod it to mode 600 before it will work
console.info                                   /var/log/console.log
```
This will make sure the console messages are logged, making it a little easier to diagnose after the fact.


----------



## VladiBG (Jul 10, 2020)

Yes i just going to said that you can log the console if you don't have physical access to it


----------



## LITMK (Jul 11, 2020)

I installed p7 and restarted. Same thing. Here is the message:


```
igb1: link state changed to UP
ums0 on uhub1
ums0: <Mouse Interface> on usbus2
ums0: 3 buttons and [Z] coordinates ID=0
uhid0 on uhub4
uhid0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 2> on usbus5
device_attach: ioapic1 attach returned 6
bridge0: Ethernet address: 02:92:6d:7b:c6:00
bridge0: changing name to 'vm-public'
igb1: link state changed to DOWN
vm-public: link state changed to UP
igb1: promiscuous mode enabled
igb1: link state changed to UP
```

In the meantime I've enabled logging of console messages. And I'll change the cable and port to check that.

When I first got  this board igb0 was doing this up/down dance so I switched over to igb1. I think Dice might be right about the card being on it's last legs. Maybe time to get a 10G! 

Thanks to everyone who has been helping! I'll let you know what I find. Very much appreciated.


----------

