# Setfib routing inside jail not working with VPN



## incorporeal (Oct 30, 2016)

I have referenced several posts and guides from Google searches pertaining to "freebsd setfib openvpn" and have had no success.

My goal is to have multiple OpenVPN connections running, and have each one run on a different routing table. I have already added the necessary net.fibs=5 to my /boot/loader.conf and rebooted my server.

Each VPN connection is represented by a different routing table. My default routing table is not connected to a VPN, so anything not in a jail just goes through my ISP. I only want jails to be routed through the VPN connection.

It would look something like this

fib 0 - Normal routing through my ISP
fib 1 - VPN connection 1 - Dallas
fib 2 - VPN connection 2 - Florda
fib 3 - VPN connection 3 - Los Angeles

From the jail perspective:

jail1.example.com - fib1 - Dallas
jail2.example.com - fib2 - Florida
jail3.example.com - fib3 - Los Angeles

I am unable to get even one jail to have internet access properly. Here is what I have done so far.

First I added a default route to my setfib 1. This is so OpenVPN can connect, if I don't set this, then it won't connect at all.


```
setfib 1 route add default 192.168.0.1
```

Next I connect to my VPN

```
setfib 1 openvpn /path/to/vpn.ovpn
```

It connects but runs into some sort of hiccup that only occurs when doing it on any setfib that is not 0.


```
Sat Oct 29 17:49:30 2016 /sbin/route add -net 10.4.0.0 10.4.30.13 255.255.0.0
route: writing to routing socket: Network is unreachable
add net 10.4.0.0: gateway 10.4.30.13 fib 1: Network is unreachable
Sat Oct 29 17:49:30 2016 ERROR: FreeBSD route add command failed: external program exited with error status: 1
Sat Oct 29 17:49:35 2016 /sbin/route add -net <EXTERNAL VPN IP> 192.168.0.1 255.255.255.255
add net 199.241.147.34: gateway 192.168.0.1 fib 1
Sat Oct 29 17:49:35 2016 /sbin/route add -net 0.0.0.0 10.4.0.1 128.0.0.0
add net 0.0.0.0: gateway 10.4.0.1 fib 1
Sat Oct 29 17:49:35 2016 /sbin/route add -net 128.0.0.0 10.4.0.1 128.0.0.0
add net 128.0.0.0: gateway 10.4.0.1 fib 1
Sat Oct 29 17:49:35 2016 Initialization Sequence Completed
```

Despite the error here, the VPN still works outside the jail. Even if I add the network manually with the command below, my results are the same in the end, the jail networking fails to use the VPN.


```
setfib 1 route add -net 10.4.0.0 10.4.30.13 255.255.0.0 -iface
```


On the HOST, I verify that the VPN is working by testing commands using setfib 1.


```
setfib 1 host google.com
google.com has address 74.125.68.101
```

and also using icanhazip.com to verify that it is my VPN IP.


```
setfib 1 curl icanhazip.com
<EXTERNAL VPN IP>
```

In the jail

This is perhaps where I'm making an error. I am assigning an ip on the cloned loopback as well an address on em0 (my only nic).


```
...
export jail_jail1_example.com_ip="lo1|127.0.1.50,em0|192.168.0.230"
export jail_jail1_example.com_fib="1"
...
```

From inside the jail I verify that I am using the correct fib


```
root@jail1:~ # sysctl net.my_fibnum
net.my_fibnum: 1
```

I test connectivity from inside the jail and I am getting nothing. I installed curl in the jail previously.


```
curl icanhazip.com
curl: (6) Could not resolve host: icanhazip.com
```

and


```
# host google.com
;; connection timed out; no servers could be reached
```


If I disconnect OpenVPN on the host and try inside the jail once more, it has internet access.


```
# host google.com
google.com has address 216.58.193.206

# curl icanhazip.com
<MY ISP IP, NOT THE VPN>
```

So basically, the VPN works on the host if I run commands with setfib 1. The VPN does not work inside the jail, and it is running on setfib1. If I disconnect the VPN on the host, the jail has functional internet on setfib1 but just through my ISP and not the VPN.

Have I missed something?

Edited: Formatting of code boxes was messed up


----------



## incorporeal (Oct 30, 2016)

Figured it out by setting the IP of the vpn client - https://blog.feld.me/posts/2015/06/routing-a-freebsd-jail-through-openvpn/ - setting it manually passed it to the jail 

One downside is that if you use the inherit option it inherits ALL the ip's from the system so if your VPN fails then you have internet through the ISP


----------



## incorporeal (Oct 30, 2016)

The networking inside the jail through the VPN seems to be rather poor. Doing a ping to google in the jail is upwards to 200MS

Inside the JAIL 

```
ping google.com
PING google.com (74.125.68.139): 56 data bytes
64 bytes from 74.125.68.139: icmp_seq=0 ttl=43 time=201.558 ms
64 bytes from 74.125.68.139: icmp_seq=1 ttl=43 time=201.931 ms
```

Doing a ping outside the jail on setfib1 (the vpn)
On the HOST

```
setfib 1 ping google.com
PING google.com (74.125.201.138): 56 data bytes
64 bytes from 74.125.201.138: icmp_seq=0 ttl=45 time=65.253 ms
64 bytes from 74.125.201.138: icmp_seq=1 ttl=45 time=63.952 ms
64 bytes from 74.125.201.138: icmp_seq=2 ttl=45 time=64.137 ms
```

Why is the performance so crappy from inside the jail?


----------

