# Multiple NICs and Jail Routing



## darrendavid (Apr 25, 2017)

Vesper's two NICs are configured as such:

_igb0:_10.0.2.20
_igb1:_ 10.0.3.20  
My plan is to host some normal services on the base OS (mostly things that need read/write access to large fiel trees, media mostly) and configure my jail to host things that will ultimately run through a VPN. So, 10.0.2.20/SERVERLAN services are accessible through inbound port forwarding from the WAN IP, and 10.0.3.21 services do their thing over a VPN. Given that, I want jails to run on _igb1/JAILLAN_, in particular, one jail at 10.0.3.21, so I can just run all that traffic through the VPN and isolate it from other networks.

Now, I can't seem to isolate the jail traffic to _igb1/JAILLAN_. When I watch it flowing through pfSense, all of the _JAILLAN _traffic is running through _SERVERLAN_. This makes me think that Vesper isn't properly sending the 10.0.3.x packets out the right interface.

I've read about fib as the way to handle multiple interfaces on FreeBSD, and I have the following bits in my _rc.conf_ on Vesper:


```
# SERVERLAN
ifconfig_igb0="inet 10.0.2.20 netmask 255.255.255.0"
defaultrouter="10.0.2.1"

#JAILLAN
ifconfig_igb1="inet 10.0.3.20 netmask 255.255.255.0 fib 1"
static_routes="jail_if jail_gw"
route_jail_if="-net 10.0.3.0/24 -iface igb1 -fib 1"
route_jail_gw="default 10.0.3.1 -fib 1"
```
The relevant bits from _/etc/jail.conf_ look like:


```
webjail {
    host.hostname = "webjail";
    path = "/jailz/webjail";
    exec.fib = "1";
    interface = "igb1";
    ip4.addr += "igb1|10.0.3.21/24";
    allow.raw_sockets;
}
```
And for purposes of completeness:


```
> setfib 0 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.2.1           UGS        igb0
10.0.2.0/24        link#1             U          igb0
10.0.2.20          link#1             UHS         lo0
127.0.0.1          link#3             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
[INDENT][/INDENT]
> setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.3.1           UGS        igb1
10.0.3.0/24        link#2             U          igb1
10.0.3.20          link#2             UHS         lo0
10.0.3.21          link#2             UHS         lo0
127.0.0.1          lo0                UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               lo0                           UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
ff02::/16                         ::1                           UGRS        lo0
```
Here's an example from the pf log on pfSense when I try to ping 10.0.3.1 from within _webjail:

[BLOCK] Apr 25 11:37:03 SERVERLAN   10.0.3.21   10.0.3.1 ICMP_

So you can see traffic that _should _be on JAILLAN is ending up on SERVERLAN. I'm stumped as to whether my issue is on Vesper or in pfSense (leaning towards the former).

Also, if I run _netstat -rn _inside the jail, I get an empty response of "Routing tables" with nothing else following. But traffic is indeed heading out, just through the wrong interface.

Thanks in advance for any and all insight here!

Darren
​


----------



## darrendavid (Apr 25, 2017)

getopt said:


> If a packet made it into the logs of pfSense you can be sure that it is blocked there.



Indeed, it's getting blocked there, but that's not the issue. The issue is that the packet from _JAILLAN _is coming through on _SERVERLAN _- I'm trying to figure out how to get it to stay on _JAILLAN _so I can set up the proper rulesets. My hunch is that something's up with my routing tables on _Vesper_, which is why packets from 10.0.3.0/24 are ending up on 10.0.2.0/24, but I'm not sure how to troubleshoot or fix beyond what I've tried so far.


----------

