# Using second NIC as FIB on different subnet



## Woden27 (Jan 11, 2022)

I configured rc.conf to support two interfaces with FIB


```
ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192 fib 0"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0 fib 1"
static_routes="Nic0 Nic1"
route_Nic0="default 203.1.1.1 -fib 0"
route_Nic1="default 192.168.20.1  -fib 1"
```

and the /boot/loader.conf


```
sysctl net.fibs
net.fibs: 2
```

ipfw  added


```
00110 setfib 0 ip from any to any via igb0
00120 setfib 1 ip from any to any via igb1
```

if I try to ping from the machine itself with


```
$ setfib 1 ping 192.168.20.2
PING 192.168.20.2 (192.168.20.2): 56 data bytes
64 bytes from 192.168.20.2: icmp_seq=0 ttl=64 time=0.102 ms
64 bytes from 192.168.20.2: icmp_seq=1 ttl=64 time=0.084 ms
64 bytes from 192.168.20.2: icmp_seq=2 ttl=64 time=0.077 ms
```

Is pinging . But if I try to ssh to that address it's going in timeout..

`$ setfib 1 ssh 192.168.20.2`


```
$setfib 1 netstat -r4
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.20.1       UGS        igb1
localhost          link#5             UHS         lo0
192.168.20.0/24    link#2             U          igb1
```

What I'm missing here?


----------



## SirDice (Jan 11, 2022)

Woden27 said:


> But if I try to ssh to that address it's going in timeout..


Firewall on the destination host blocking the traffic?


----------



## Woden27 (Jan 11, 2022)

the destination host is the second NIC itslef


----------



## SirDice (Jan 11, 2022)

Oh, right. I see that now. Is sshd(8) actually listening on that address? What does `sockstat -4` tell you?


----------



## covacat (Jan 11, 2022)

probably listens otherwise it would be connection refused
but ssh replies from fib0 and will send to 192.168.x.x thru the internet which will fail


----------



## Woden27 (Jan 11, 2022)

SirDice said:


> Oh, right. I see that now. Is sshd(8) actually listening on that address? What does `sockstat -4` tell you?


`root     sshd       3960  5  tcp4   192.168.20.2:22       *:*`


----------



## Woden27 (Jan 11, 2022)

covacat said:


> probably listens otherwise it would be connection refused
> but ssh replies from fib0 and will send to 192.168.x.x thru the internet which will fail


Probably something like this, but how I can solve it?


----------



## VladiBG (Jan 11, 2022)

I don't know what is your end goal for this but if it's to have redundancy WAN access then you need dynamic routing or if it's not possible then go for SLA tracking.


----------



## Woden27 (Jan 11, 2022)

VladiBG said:


> I don't know what is your end goal for this but if it's to have redundancy WAN access then you need dynamic routing or if it's not possible then go for SLA tracking.


Just I would like to reach another network from the NIC interface igb1...nothing special !
I am trying to test the connectivity, but I cannot reach the 192.168.20.0/24 in any way..neither from the same machine and the same interface. (but when attached to the other router is the same -- timeout)


----------



## covacat (Jan 11, 2022)

you can probably use ipfw to set fib 1 on any packet originating on 192.168.20.2


----------



## Woden27 (Jan 11, 2022)

covacat said:


> you can probably use ipfw to set fib 1 on any packet originating on 192.168.20.2




```
00100 allow ip from any to any via lo0
00110 setfib 0 ip from any to any via igb0
00120 setfib 1 ip from any to any via igb1
00125 setfib 1 ip from any to any via 192.168.20.2
```

even if I delete rule 00120 -- same
even if I disable ipfw -- same


----------



## VladiBG (Jan 11, 2022)

Woden27 said:


> Just I would like to reach another network from the NIC interface igb1


Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.


----------



## Woden27 (Jan 11, 2022)

VladiBG said:


> Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.


I changed to
`route_Nic1="-net 192.168.20.0/24 192.168.20.1  -fib 1"`

again, I can ping the NIC address , but when I try to ssh it , timeout...

ipfw service is stopped...and sshd is listening !

`root     sshd       1728  5  tcp4   192.168.20.2:22       *:*`


----------



## VladiBG (Jan 11, 2022)

sshd is running on fib0


----------



## Woden27 (Jan 11, 2022)

VladiBG said:


> Then you can use static route to that network via igb1. You don't need multiple FIB not/or two default gateways.


OK i solved removing all the FIB statements  and using the static route for igb1.
This works, but anyway, I would like to know why with FIBs fails?


----------



## Woden27 (Jan 11, 2022)

VladiBG said:


> sshd is running on fib0


I even added these statements to sshd_config ...

`ListenAddress 0.0.0.0
ListenAddress 192.168.20.2`


----------



## VladiBG (Jan 11, 2022)

The igb1 has 192.168.20.2/24 so you don't need to define a static route for it via 192.168.20.1 as it's connected route and this gateway will never be used to reach rest of the hosts in 192.168.20.0/24.

in sshd


> ListenAddress 0.0.0.0


means that it will listen on all IPs. In configuration files only the last statement is apply as it's overwrite the previous one. So specifying multiple times the same statement make no sense.

If you are interested in networking i recommend you to start with CCNA Routing and Switching Lessons


----------



## Woden27 (Jan 11, 2022)

> The igb1 has 192.168.20.2/24 so you don't need an to define a static route for it via 192.168.20.1 as it's connected route and this gateway will never be used to reach rest of the hosts in 192.168.20.0/24.


This makes sense  to me.



> means that it will listen on all IPs. In configuration files only the last statement is apply as it's overwrite the previous one. So specifying multiple times the same statement make no sense.



even this sounds ok.

The fact I was confused -- and the reason I used FIBs was actually the default router statement.
If my configuration is:

`ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0"
defaultrouter="203.1.1.1"`

How the traffic from the interface igb1 can flow without a "default router" statement or a static route?


This doesn't solve the problem with FIBs and why in that case the address is pingable but not sshable ..


----------



## VladiBG (Jan 11, 2022)

Woden27 said:


> How the traffic from the interface igb1 can flow without a "default router" statement or a static route?


It receive directly connected route for 192.168.20.0/24 via igb1. Same as directly connected route for 203.1.1.0/26 via igb0. Those subnets are accessible without routing the traffic trough the gateway/router/.


----------



## covacat (Jan 11, 2022)

if you want all traffic from from 192.168.20.2 to go to internet thru 192.168.20.1
add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1
add allow ip from 192.168.20.2 to me 
add fwd 192.168.20.1 ip from 192.168.20.2 to any
no fibs
ping -S 192.168.20.2 8.8.8.8
or you can probably run multiple instances of sshd or any other services and bind them separately on igb0 and igb1 and use fibs


----------



## Woden27 (Jan 12, 2022)

VladiBG said:


> It receive directly connected route for 192.168.20.0/24 via igb1. Same as directly connected route for 203.1.1.0/26 via igb0. Those subnets are accessible without routing the traffic trough the gateway/router/.


Actually removing the statement 

`defaultrouter="203.1.1.1"`

will make the server accessible from the LAN but not from the WAN with PAT from the main router.


----------



## Woden27 (Jan 13, 2022)

covacat said:


> if you want all traffic from from 192.168.20.2 to go to internet thru 192.168.20.1
> add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1
> add allow ip from 192.168.20.2 to me
> add fwd 192.168.20.1 ip from 192.168.20.2 to any
> ...


Tried, I can ping the default GW (192.168.20.1) but cannot reach the server to 192.168.20.2 via ssh
I cannot ping outside on the internet  neither from the interface igb1
It just goes in timeout.

This is what I have at the moment in my rc.conf

`ifconfig_igb0="inet 203.1.1.7 netmask 255.255.255.192"
ifconfig_igb1="inet 192.168.20.2 netmask 255.255.255.0"
defaultrouter="203.1.1.1"
static_routes="Nic1 2df"
route_Nic1="-net 192.168.20.0/24 -iface igb1"
route_2df="default 192.168.20.1 -iface igb1"`

i addedd the rules suggested with

`ipfw (role number after all the allow statements)  add allow ip from 192.168.20.0/24 to 192.168.20.0/24 via igb1`

etc..


----------



## covacat (Jan 13, 2022)

you have to have some kind of nat on 192.168.20.1 in order to access internet from 192.168.20.2


----------



## Woden27 (Jan 13, 2022)

Nat is working correctly.
To prove that i just connected a laptop with windows  with the same address (static) to the same port in the same VLAN, and i can ping google without problems..

must be something else, but honestly I have now no idea


----------



## covacat (Jan 13, 2022)

```
box1 connected to internet via ng0 (pppoe)
10.1.1.1 is on internal lan and gateway for the rest of the lan

~#netstat -rn|grep defau
default            10.0.0.1           UGS         ng0


box2 has 10.1.1.42 and an openvpn connection to another host
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
        inet 10.255.255.10 --> 10.255.255.9 netmask 0xffffffff

on box2
route add 11.12.13.14/32 10.255.255.9  (send traffic to 11.12.13.14 thru openvpn)
bash-3.2# netstat -rn |grep 11.12.13.14
11.12.13.14/32     10.255.255.9       UGSc         utun4      

on box1
ipfw add 5 fwd 10.1.1.42 icmp from 10.1.1.1 to 11.12.13.14
send icmp traffic from 10.1.1.1 to box2
ping -S 10.1.1.1 11.12.13.14
PING 11.12.13.14 (11.12.13.14) from 10.1.1.1: 56 data bytes

on box2
bash-3.2# tcpdump -i utun4 -nl
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on utun4, link-type NULL (BSD loopback), capture size 262144 bytes
13:17:56.527595 IP 10.1.1.1 > 11.12.13.14: ICMP echo request, id 38695, seq 11, length 64
13:17:57.531496 IP 10.1.1.1 > 11.12.13.14: ICMP echo request, id 38695, seq 12, length 64

so traffic sent from box1 10.1.1.1 to 11.12.13.14 is going thru the openvpn connection
no routes have been set on box1 just ipfw fwd
```

box1 is freebsd 12.3 box2 is a mac


----------



## tanis (Jan 13, 2022)

My first take on something like that is always traceroute. Did you try that locally and also from remote ?

Did you try to capture what's happening, when u try to connect to your local interface 192.168.20.2 using tcpdump ?

```
$ tcpdump -i igb1 host 192.168.20.2 and port ssh
```

Same goes for, did you try to reach 192.168.20.2:ssh remotely, for instance from 192.168.20.1 ?

What is the bigger picture of this configuration ? You mentioned the host is doing NAT and there is also a VLAN involved. Usually NAT indicates that your host is a router and if a router is connected to a VLAN the port is in trunk mode, which means all packages are tagged by the switch with their appropriate VLAN ID, which also means that there  usually needs to be some configuration on the router as well. You also mentioned WAN which usually indicates where the default route is pointing, on the other hand you want that there is a second "default" route to a network behind 192.168.20.1, which brings me back to my initial question for this paragraph. 

The last thing for debugging purpose, I would recommend, is to disable the firewall. 

Edit:
I forgot to mention regarding your sshd configuration. There are four options here:
Option A)
Listen 0.0.0.0

Option B)
Listen 192.168.20.2
Listen 203.1.1.7

Option C)
Listen 192.168.20.2

Option D)
Listen 203.1.1.7

Option A and Option B are the same. A combination of Option A) with any of the other options (B, C, D), I cannot  recommend.


----------

