# Multiple nics and setfib



## Understudy (Apr 1, 2016)

Hi All,

I am looking to setup two physical nics with two different ip addresses and two different gateways. The basics:


```
10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015  
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386
```

I am looking to make one nic part of the private ip range 192.168.1.0/24 with a gateway of 192.168.1.255. I am looking to make the second nic part of a public ip range 77.77.77.0/24 with a gateway of 77.77.77.77 . As I am attempting to understand I need to setup setfib for this to work properly. My understanding is this has become part of the kernel default after version 10. 

So I am looking to place a line in /boot/loader.conf.


```
## Loader Config ##
net.fibs=2
```

Then modify my /etc/rc.conf


```
## RC.CONF ##
static_routes="fibdefault"
route_fibdefault="default 192.168.1.255 -fib 1
route_fibdefault="default 77.77.77.77 -fib 2
ifconfig em0="DHCP"
ifconfig em1="77.77.77.76 netmask 255.255.255.240"
```

How does this look? Thank you in advance.


----------



## usdmatt (Apr 1, 2016)

```
route_fibdefault="default 192.168.1.255 -fib 1
route_fibdefault="default 77.77.77.77 -fib 2
```
You definitely can't do that for a start. /etc/rc.conf is just a list of variables, so route_fibdefault is set to the first value, then you overwrite it with a different value in the second line.

First off, do you actually need multiple gateways. Is it a problem if all access to non local addresses (anything that isn't 192.168.1.x) goes via 77.77.77.77? If not, the easiest solution is just to set a static 192 address on em0, and give a default gateway of 77.77.77.77.

```
defaultrouter="77.77.77.77"
ifconfig_em0="192.168.1.x/24"
ifconfig_em1="77.77.77.76/28"
```
 That's what I'd do unless there was a really specific reason why this machine couldn't just use the 77 address to access the Internet.

If you do want multiple routing tables, you should be able to use the default options and just manually set the default route for the second routing table.

```
ifconfig_em0="DHCP"
ifconfig_em1="77.77.77.76 netmask 255.255.255.240"
```
Hopefully DHCP should set your main default gateway. It's then just a case of adding the gateway for the second routing table

```
# setfib 1 route add default 77.77.77.77
```
There may be a way of doing this via rc.conf although 60 seconds on Google didn't find anything obvious. Most results seem to suggest putting that command in /etc/rc.local. If it complains about the route already existing, which it shouldn't really, you can use change instead of add, or delete the route first:

```
setfib 1 route delete default
```
Bear in mind I haven't tested any of this, or ever used multiple routing tables.

Also, all applications will use the original routing table by default, and go via 192.168.1.255*. You'll have to actually run applications using `setfib 1` for them to use the second routing table.

*Also 2, If 192.168.1.255 is the end of your range, you shouldn't be using it on a device. Assuming you're using the most common network size (subnet mask 255.255.255.0), the range would be 192.168.1.0-192.168.1.255, and 255 is the broadcast address.

Edit: I've just read the route man page a bit more and seen the fib stuff. So, the following *should* be feasible

```
ifconfig_em0="DHCP"
ifconfig_em1="77.77.77.76 netmask 255.255.255.240"
static_routes="gw2"
route_gw2="default 77.77.77.77 -fib 1"
```


----------



## Understudy (Apr 2, 2016)

Thank you for your reply.

I can set a static 192.168.0.X address if I need to. I just had not planned on it.
I am not sure the the 192.168.0.X address would be accessible through the 77.77.77.77 gateway. I am doing this in coordination with an AT&T modem that hands off to me both private and public IPs. The issue being that they both run through separate gateways even if it is the same modem.




usdmatt said:


> ```
> defaultrouter="77.77.77.77"
> ifconfig_em0="192.168.1.x/24"
> ifconfig_em1="77.77.77.76/28"
> ...



The 77.77.77.76 ip is used to access the internet.

I would like the office where the servers are located to be able to access the server with SSH from the internal network especially when there is a problem with the dns or other on the internet that prevents us from accessing the public IP.



usdmatt said:


> *Also 2, If 192.168.1.255 is the end of your range, you shouldn't be using it on a device. Assuming you're using the most common network size (subnet mask 255.255.255.0), the range would be 192.168.1.0-192.168.1.255, and 255 is the broadcast address.



I was just using it as an example the ISP 192.168.1.254 for the broadcast. My apologies for the confusion.



usdmatt said:


> Edit: I've just read the route man page a bit more and seen the fib stuff. So, the following *should* be feasible
> 
> ```
> ifconfig_em0="DHCP"
> ...



Would this go in the /etc/rc.conf?


----------



## Understudy (Apr 3, 2016)

Trying a couple of different things here but still not getting the desire result.
The idea being that emo is the default which would be fib 0 so you don't have to create a setfib for that.


/etc/rc.conf 

```
ifconfig_em0="DHCP"
static_routes="fibdefault"
route_fibdefault="default 77.77.77.77 -fib 1"
ifconfig em1="inet 77.77.77.76 netmask 255.255.255.240"
```

does not appear to work

`# sysctl net.fibs`

```
net.fibs: 2
```

`# setfib 1 netstat -rn`

```
Routing tables (fib: 1)
Internet:
Destination  Gateway  Flags  Netif Expire
127.0.0.1  link#3  UH  lo0
192.168.1.0/24  link#1  U  em0
Internet6:
Destination  Gateway  Flags  Netif
Expire
::/96  ::1  UGRS  lo0
::1  link#3  UH  lo0
::ffff:0.0.0.0/96  ::1  UGRS  lo0
fe80::/10  ::1  UGRS  lo0
fe80::%lo0/64  link#3  U  lo0
ff02::/16  ::1  UGRS  lo0
```

`# ifconfig`

```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b8
  inet 192.168.1.143 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active

em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b9
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
```


----------



## kp_ (Apr 3, 2016)

Try to put the second NIC (em1) into the fib 1 _and_ the default route.


----------



## usdmatt (Apr 3, 2016)

Is it even possible to put into NIC into fib 1? As I understand it, the interfaces are static across all fibs, you just have different routing tables.



> I would like the office where the servers are located to be able to access the server with SSH from the internal network especially when there is a problem with the dns or other on the internet that prevents us from accessing the public IP.



You realise that the machine is perfectly accessible via its private IP from any client that has a 192.168.1.x address, even if its default gateway is via a different interface?


----------



## Understudy (Apr 4, 2016)

usdmatt said:


> You realise that the machine is perfectly accessible via its private IP from any client that has a 192.168.1.x address, even if its default gateway is via a different interface?



It is an either or situation. I can either have it as a private ip and accessible via ssh from within the office or I can have a public ip and it is accessible from both as long as the DNS or something else doesn't flub up. I have had it where I cannot access it via the public ip because of outside issues. I would like to be able to at least have a type of solution on that.


----------



## Understudy (Apr 4, 2016)

Some work I have been doing:

Current /etc/rc.conf


```
hostname="postal.brendhanhorne.com"
ifconfig_em0="DHCP"
default_router="192.168.1.255"
ifconfig_em1="up"
static_routes="fibnet fibgate"
route_fibnet="-net 77.77.77.76/28 -interface em1 -fib 1"
route_fibgate="default 77.77.77.77 -fib 1"
sshd_enable="YES"
moused_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
# SSH
sshd="YES"
```

Here are the results of the `ifconfig`

```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b8
  inet 192.168.1.143 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b9
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

And then I end up with a numbers that are off with a netstat command.
it say 77.77.77.74. It shouldn't. I can't `ping` it either.

`# setfib 1 netstat -rn`

```
Routing tables (fib: 1)

Internet:
Destination  Gateway  Flags  Netif Expire
default  77.77.77.74  UGS  em1
77.77.77.74/28  00:30:48:87:32:b9  US  em1

Internet6:
Destination  Gateway  Flags  Netif Expire
fe80::%lo0/64  link#3  U  lo0
```

So basically nothing has changed. And I am getting information from a `netstat` command that only helps to confuse.


----------



## usdmatt (Apr 4, 2016)

> It is an either or situation. I can either have it as a private ip and accessible via ssh from within the office or I can have a public ip and it is accessible from both as long as the DNS or something else doesn't flub up. I have had it where I cannot access it via the public ip because of outside issues. I would like to be able to at least have a type of solution on that.



It isn't an either or situation though. If you have a public IP and gateway on the server it's accessible from the Internet. If you then add a private interface on your LAN, it's accessible from the LAN using the 192 address, even if the public interface is completely down; It might just take a few seconds longer to connect because the DNS lookup will fail.

I don't think a small ssh delay is worth the hassle of trying to engineer two gateways (something that is nowhere near as common as people seem to think), which will probably make no difference because the dns resolver is not going to somehow know which routing table it needs to use to access a dns server; It will just use the routing table the application (i.e ssh) was set to use when it was started.


----------



## kp_ (Apr 5, 2016)

usdmatt said:


> Is it even possible to put into NIC into fib 1? As I understand it, the interfaces are static across all fibs, you just have different routing tables.




```
# ifconfig vmnet0
vmnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:bc:37:00:00
        inet 10.255.255.3 netmask 0xfffffffe broadcast 255.255.255.255
        nd6 options=8<IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
        fib: 3
```
See man ifconfig:


> fib fib_number
> Specify interface FIB.  A FIB fib_number is assigned to all
> frames or packets received on that interface.  The FIB is not
> inherited, e.g., vlans or other sub-interfaces will use the
> ...


----------



## olivier (Apr 5, 2016)

Small examples:

```
ifconfig_vtnet0="10.0.12.1/24 fib 2"
ifconfig_vtnet1="10.0.13.1/24 fib 3"
ifconfig_vtnet2="10.0.14.1/24 fib 4"
static_routes="ISP1 ISP2 ISP3"
route_ISP1="-fib 2 default 10.0.12.2"
route_ISP2="-fib 3 default 10.0.13.3"
route_ISP3="-fib 4 default 10.0.14.4"
```


----------



## magikman (Apr 5, 2016)

Understudy said:


> Hi All,
> 
> I am looking to setup two physical nics with two different ip addresses and two different gateways. The basics:
> 
> ...



I've never worked with fib before but I can tell you that you are trying to set the gateway to the broadcast address of that subnet. .254 is the last usable address in that range.


----------



## Understudy (Apr 5, 2016)

magikman said:


> I've never worked with fib before but I can tell you that you are trying to set the gateway to the broadcast address of that subnet. .254 is the last usable address in that range.


Yes, that was me having a brain fart. My apologies for the confusion.


----------

