# Jail networking



## perkypork (Mar 5, 2015)

I am using ezjails to create jails on a Dell R710 with 4 network interfaces. 1 nic is being used for basejail traffic and the remaining 3 nics are being used for different types of jails. All 3 nics are connected to the public internet. I am splitting the jails over 3 nics to maximise throughput.

How in this scenario am I able to use all 3 jail nics for traffic? It seems like I am going to have to configure multiple gateways. Is someone able to offer any alternative options available to me?


----------



## perkypork (Mar 6, 2015)

Not really looking to use lagg because I loose some flexibility i.e. I cannot use different subnets on different interfaces, etc. Thanks for giving me a pointer though.


----------



## perkypork (Mar 10, 2015)

To add further information to this post. I am using Jails for this server.

I have 4 NICs on my server. I use one nic for base system management. I would like to use of the other 3 nics with potentially a different "gateway". When I say gateway, I don't think I mean it in the technical sense, I mean that I want a jail to respond back via the interface its been given. I have done a ton of reading about this but I am stuck. I don't want to use setfib(1) unless there is no other way but I don't understand static routes properly.

Maybe if I give some examples of something I would like to achieve.

One of my base systems has an IP of 192.168.123.192. My VPN gives me an IP of 10.0.123.2. The default gateway on the base system is a public IP. I would like to be able to access the base system IP from my VPN connection. Currently I the base system does not respond to any traffic from the 10.0.123.0/24 subnet. Am I able to get this working via static routes for the base system interface?

I would like to create a jail that has an IP on the local network - 192.168.123.45 but using a different interface than the base system. The only way I can access the local network is via a VPN which hands out IPs from the 10.0.123.0/24. The default gateway on the base system is a public IP. Am I able to get this working via static routes for the base system interface?

If I am able to configure the above via static routes could someone please explain how I would do this in rc.conf?


----------



## asteriskRoss (Mar 10, 2015)

Hi perkypork. I'm a bit confused about your network architecture and where your VPN fits in.  A sketch diagram might help.



perkypork said:


> I am using ezjails to create jails on a Dell R710 with 4 network interfaces. 1 nic is being used for basejail traffic and the remaining 3 nics are being used for different types of jails. All 3 nics are connected to the public internet. I am splitting the jails over 3 nics to maximise throughput.


It sounds like the four interfaces on your server are connected to the same physical network.  Is that correct?



perkypork said:


> My VPN gives me an IP of 10.0.123.2.


What is the "me" in this description? An interface on the server or a client machine?  Where is the VPN endpoint in relation to the network the server is on?

Information that may, or may not be useful: Jails (including those created using sysutils/ezjail) can be configured with multiple interfaces.  Have a look at the excellent ezjail section of the FreeBSD handbook (written by wblock@).  So, if you wanted two jails on totally different networks to be able to communicate without routing traffic externally, you could configure a shared local interface in addition to the external interfaces.

bthomson is right to suggest lagg(4).  If you want to maximise throughput, you could aggregate all your interfaces together using lacp, loadbalance or roundrobin configuration and then assign the jails addresses on the aggregated interface (which you might call lagg0).  The aggregation means that if one of your jails is generating a lot of network traffic but the other two are quiet, the traffic will be split across all the interfaces.  If you do as you suggest and just assign a jail an address on one interface (without aggregation), that interface could be very busy whilst your other interfaces sit doing nothing.


----------



## perkypork (Mar 10, 2015)

asteriskRoss said:


> Hi perkypork. I'm a bit confused about your network architecture and where your VPN fits in.  A sketch diagram might help.
> 
> It sounds like the four interfaces on your server are connected to the same physical network.  Is that correct?



My question has changed a little since I first posted. What I would like to do now is have the ability for each interface to be connected to a different network and have the jails be able to respond to traffic on their specific IP/network.

My OpenVPN connection is a pfsense firewall with a tunnel network of 10.0.123.0/24 which pushes a route to the 192.168.123.0/24 network. The OpenVPN connection is the only way I can access this system remotely. The private network where this server resides is 192.168.123.0/24, the base system has an IP in this range. If I leave the default gateway on the base system to be the gateway of the local network (192.168.123.0/24) then I can access the base system from the VPN. The only problem with doing this is any jails with a public IP stop responding to traffic. If I change it so that the default gateway is one of our public gateways then I am unable to access the base system on its private IP from my VPN connection. I would to know how to configure FreeBSD so traffic is responded to on all interfaces, not an OR scenario like I currently have.

I will do a drawing up when I get to the office.


----------



## perkypork (Mar 10, 2015)

I decided that I would try this out on one of our systems instead of trying to understand everything before configuring it. I added a static route to see if it would work:

`route add -net 10.0.123.0/24 192.168.123.254`

It started working, YEY!

The reason I was asking about this is I have found it hard to find information about this that comes from the angle of "What I want to achieve" instead of the technical.

For someone who might be looking for more of a practical example. By adding the above static route to my system, when traffic comes from my VPN connection, the system knows where to respond to it. Previously it would receive the traffic from 10.0.123.0/24 on its local IP 192.168.123.192 and respond via the defaultrouter (a public IP), which was never going to work. Adding the static route meant that instead of replying via the defaultrouter it now responded via 192.168.123.254. 192.168.123.254 is the address of our pfsense system and that system knows how to communicate with the VPN network 10.0.123.0/24. Please correct me if I am wrong.


----------



## perkypork (Mar 11, 2015)

I think I jumped the gun.

I added the static route above and while I was now able to access any server on the local network (192.168.123.0/24) from the VPN, none of those servers could access the internet. To solve this, I switched things around and made 192.168.123.254 the defautrouter (our local gateway) and added a static route to our WAN gateway. This seems to work for those servers with a public IP providing HTTP/HTTPS traffic but SSHing to any of these servers fails.

I am unsure how to solve this issue, any help would be appreciated!


----------



## asteriskRoss (Mar 11, 2015)

perkypork, I see you have also started Thread jails-configuration-questions.50387 and Thread dual-gateways.49126, where you appear to be tackling the same issue.  Perhaps it would be sensible to consolidate these?


----------



## perkypork (Mar 11, 2015)

asteriskRoss said:


> perkypork, I see you have also started Thread jails-configuration-questions.50387 and Thread dual-gateways.49126, where you appear to be tackling the same issue.  Perhaps it would be sensible to consolidate these?



This is a good question asteriskRoss. One of the reasons I didn't continue asking questions in those posts is that I didn't realise they were related. I have struggled finding information on how to achieve what I want to achieve. Much of the documentation I have found discusses things in a technical way, which requires that you know how you want to achieve something - being a newbie I have no idea how to achieve something, only what I want to achieve.

This post jails-configuration-questions.50387 was about the best way of implementing jails.

This post dual-gateways.49126 was about understanding the differences between Linux and FreeBSD networking. Coming from the Linux world, I was unaware that multiple gateways are not required in most scenarios. In my Linux days, I would just set a gateway for each interface and be done with it.

The current post jail-networking.50639 was about understanding how I should configure my networking in jails. This post and the dual gateways post could be combined but then newbies who use these questions to work out what they want to achieve may be confused by the combination of migration from Linux (centos) and Jails networking.


----------



## kpa (Mar 11, 2015)

perkypork said:


> This post dual-gateways.49126 was about understanding the differences between Linux and FreeBSD networking. Coming from the Linux world, I was unaware that multiple gateways are not required in most scenarios. In my Linux days, I would just set a gateway for each interface and be done with it.



I don't remember how Linux handles gateways but in FreeBSD there is no direct connection between interfaces and gateways in the routing table, you can't "set a gateway" for an interface because it just doesn't make sense in any way. Instead the gateways are always tied to routes in the routing table.


----------



## asteriskRoss (Mar 12, 2015)

perkypork, a network diagram of what you want to achieve would still be helpful   It doesn't need to be pretty!


----------



## perkypork (Mar 13, 2015)

asteriskRoss said:


> perkypork, a network diagram of what you want to achieve would still be helpful   It doesn't need to be pretty!



asteriskRoss - its not pretty.

I now have everything working except the local jails. When I try to install a port, the jail doesn't know how to get on the net. This is because my defaultrouter is a public IP. I have put static routes for the VPN networks on pfsense, which means we can access the local jails and the base system from the VPN. The public jails have no issues so far.

All that is left is getting the jails with a local ip to be able to get ports.


----------



## asteriskRoss (Mar 13, 2015)

I'm pleased you have most of what you want working.  If you need any further input on the routing side it would be great if you could post the routing tables (`netstat -r`) as well as the interface configuration (`ifconfig`) on your server and the pfSense box so I and other forum members can make sure we understand your set up.

Regarding the jails connected to your local network, if you don't have public IPv4 addresses to spare and you're using private IPv4 addresses (you indicated they are on 192.168.123.0/24), you'll need to configure Network Address Translation (NAT) to present a routable IP address to the public Internet.  You could do this on your server (in conjunction with a firewall, using your server as a router) or using the pfSense box as your router.

An alternative (not hassle-free) would be to host your own package repository on the server, which might be more efficient since otherwise you may be downloading and compiling the same ports multiple times.  If this is the way you choose, I highly recommend ports-mgmt/poudriere (see also the section in the FreeBSD handbook).  Once you have your repository built, you could share it among your jails by using nullfs(5) to mount the directory tree in each jail or even set up a jail with a webserver on your local network to serve the package repository over HTTP.


----------



## asteriskRoss (Mar 13, 2015)

I should add that if you're using binary packages rather than ports and you just have a couple of packages to install in a jail, an easy option is to obtain the packages and install them from disk rather than over the network with `pkg add <package_name>.tgz`. This can even be done with ports-mgmt/pkg itself.


----------



## perkypork (Mar 13, 2015)

asteriskRoss said:


> I should add that if you're using binary packages rather than ports and you just have a couple of packages to install in a jail, an easy option is to obtain the packages and install them from disk rather than over the network with `pkg add <package_name>.tgz`. This can even be done with ports-mgmt/pkg itself.



Thanks asteriskRoss!

I am currently building from source. Ideally I would like to use poudriere but at the moment, I am holding that on the backburner until I get my jails stuff sorted out. I have attached my routing table and network setup.


----------



## perkypork (Mar 13, 2015)

I have not run out of public IPv4 addresses, I use the private IPs for staging servers and the public IPs for live servers. The base system has a private IP.

I VPN in to get access to the staging servers. The staging servers get internet access for ports etc through pfsense (vpn) box.


----------



## perkypork (Mar 13, 2015)

perkypork said:


> asteriskRoss - its not pretty.
> 
> I now have everything working except the local jails. When I try to install a port, the jail doesn't know how to get on the net. This is because my defaultrouter is a public IP. I have put static routes for the VPN networks on pfsense, which means we can access the local jails and the base system from the VPN. The public jails have no issues so far.
> 
> All that is left is getting the jails with a local ip to be able to get ports.



My only (but big) issue with the above configuration is if I need to restart networking for a new IP to be read from rc.conf. I must run the command `service netif restart & service routing restart` or I am unable to access the base system from the VPN. Ideally I would have my default router to be the private IP of the pfsense (VPN) box and have a static route for the internet. This would allow me not to worry about being locked out because of a typo.


----------



## asteriskRoss (Mar 14, 2015)

*Suggestion: Configure two different routing tables with setfib(1)*
I know from your other posts you had been trying to avoid using multiple routing tables, but in this case it sounds like it might be a good option.  setfib(1) allows you to have multiple routing tables and you can specify which one you want to use for each jail (see also the man page for setfib(2) to configure options in your kernel or /boot/loader.conf).

Your production jails and staging systems (in which I'm including your host system, which you said also has an IP on the 192.168.123.0/24 network) have different routing requirements.  The default route for your production systems needs to be via your public gateway, whereas the default route for your staging systems needs to be via the pfSense box, which can then handle NAT routing to the Internet.

The ezjail configuration file has a configuration option for specifying which routing table to use (jail_${jail_name}_fib if I remember correctly), which is equivalent to setting the exec.fib parameter in jail.conf(5) for a non-ezjail jail.
*
Option involving no static routes*
With the default gateway of your staging systems being the pfSense box, I don't believe you would need static routes on either your staging systems or the pfSense box.  The pfSense box has addresses on the 10.0.123.0/24 network (for your VPN) and also the 192.168.123.0/24 network (for your staging systems), so the pfSense's routing tables will be automatically populated to route traffic between these two networks.  Anything else would be routed out via NAT to the public Internet.  I'll caveat this by saying that I don't use pfSense so I'm not sure what special configuration (if any) it requires.

With the default gateway of your production jails being your public gateway, these can respond appropriately to incoming requests without issue via this default route.  For remote access from your VPN, you could connect via ssh(1) to your host system (on the 192.168.123.0/24 network) then attach to your jails with `ezjail-admin console <jailname>` or with jexec(8).  This wouldn't require any static routes.
*
Option to allow direct SSH connections to production jails from the VPN*
If you wanted to connect with ssh(1) directly to your production jails via the VPN, a reasonable option would be to assign them an address on an interface connected to the 192.168.123.0/24 network as well as their public IP.  In my opinion it would be better to run your SSH servers listening only on the private addresses so they can be accessed via the VPN but not from the public Internet.  You would then need to configure a static route so that the production jails route traffic for the 10.0.123.0/24 network (for VPN connections) via the pfSense box.

Does that make sense?  I hope I've understood your network and requirements correctly 

_Edit: Various typo corrections_


----------



## perkypork (Mar 14, 2015)

asteriskRoss - I cannot thank you enough for helping me out with this! The best I can do is say "in the short time I have been using FreeBSD I am a huge convert, the OS is the best server OS I have used. What makes FreeBSD even better is the great community it has. People like yourself and all the other people on the forums who have helped me very patiently, is something very special for an open source project"

I will implement multiple routing tables as it seems like this is my only option. Thanks again!


----------



## junovitch@ (Mar 14, 2015)

This should get you started.  I skimmed the thread and haven't read all the details but at least wanted to throw this out there to get you rolling.
`echo 'net.fibs=4' >> /boot/loader.conf`

Set a default route on each FIB.
/etc/rc.conf

```
static_routes="fib1default fib2default fib3default"
route_fib1default="default 1.1.1.1 -fib 1"
route_fib2default="default 2.2.2.2 -fib 2"
route_fib3default="default 3.3.3.3 -fib 3"
```

Now just set your jails to use the FIB you want them to be on.  This is the exec.fib parameter in jail.conf.   Just set it to the FIB number you want and all traffic in that jail will use the default gateway you assigned on that FIB.


----------



## perkypork (Mar 15, 2015)

junovitch - Thanks! I have had a crack at implementing 2 fibs.

I tried your initial instructions and found a similar issue to below, so changed it to the below to see if the issue fixed itself. I think I like the below better because it gives a little more control but more importantly (as a newbie) I need to know what is going on instead of things happening automatically so I learn stuff.

`echo 'net.fibs=2' >> /boot/loader.conf`
`echo 'net.add_addr_allfibs=0' >> /boot/loader.conf`

I then setup my static routes (see screenshot).

Almost everything seems to be working at this point, except for the firewall. I use IPFW and the base system is now unable to ssh or ping to any jail with a public IP on the same box (permission denied). Before implementing fibs I had no issues. I have not changed the firewall rules at all. My firewall rules are in attached screenshot. I am not sure what I need to change to get it working properly.


----------



## perkypork (Mar 15, 2015)

I get the feeling my ssh (IPFW) issue has something to do with the fact that the base system can now see two routing tables and my IPFW config only allows communication from the base system from its local IP. I want each system to only communicate on its allocated IP and the base system only has a private IP.


----------



## junovitch@ (Mar 16, 2015)

When you are using `echo 'net.add_addr_allfibs=0' >> /boot/loader.conf` things get a big odd.  Since the local address don't get added that means the replies to the local host are taking a round trip to the default router which likely is configured to not send the packet back out the same interface it was received on.  If you use that tunable, try something along the lines of this to give a path back for the reply traffic that won't require the round trip to the gateway.

```
route_fib1network="-net 192.168.102.0/24 -interface em0 -fib 1"
```

Try to use tcpdump(1) at on your gateway and on lo0 to gain an insight on traffic leaving and staying local on the box.  Tweak your routes until you get the behavior you are looking for.


----------



## perkypork (Mar 16, 2015)

I have done the below, which you can see in the screenshots (above) of my configuration.



junovitch said:


> When you are using `echo 'net.add_addr_allfibs=0' >> /boot/loader.conf` things get a big odd.  Since the local address don't get added that means the replies to the local host are taking a round trip to the default router which likely is configured to not send the packet back out the same interface it was received on.  If you use that tunable, try something along the lines of this to give a path back for the reply traffic that won't require the round trip to the gateway.
> 
> ```
> route_fib1network="-net 192.168.102.0/24 -interface em0 -fib 1"
> ```


----------



## junovitch@ (Mar 16, 2015)

Ok, sorry about that.  I've looked at the pictures now.  It looks like you've got the routes for both FIBs to see each other.  Check with something like `tcpdump -i lo0 -n port 22` to see if you see the traffic floating around on the loopback and particularly if you see anything like R flags for RST's being returned.


----------



## perkypork (Mar 16, 2015)

I ran `tcpdump -i lo0 -n port 22` and got no traffic. It seems the traffic is being blocked by my firewall as I am getting a permission denied response. I only get this if I am trying to ssh into jails from the base system. I can happily SSH to any IP (public or private) not on the base system.

To explain this a little further. If you look at my routing tables, fib 0 contains routing information for the public ip range on the box. It seems that when I try to ssh from the base (which uses a private IP and has firewall rules only allowing traffic in/out for this private ip) its trying to use bce1 which its not able to do.


----------



## perkypork (Mar 16, 2015)

While I wait for some generous person to help me out, I went and reconfigured my system following junovitch initial instructions (again) for getting setfib(1) up and running. I still have the same issue. I am unable to ssh from the base system to a jail (running on the base) with a public ip. Sadly after Googling/reading for a few hours I cannot seem to find a way to fix it.


----------



## perkypork (Mar 16, 2015)

After a number of hours trying to work out how to fix my issue, it seems I have fixed it but I have no idea how. Very annoying, especially since I wanted to document what I did for others.

I have to build another one of these boxes, I will do my steps all over again and document exactly what I did.


----------



## asteriskRoss (Mar 16, 2015)

perkypork, which address(es) did you configure your SSH servers to listen on (in /etc/ssh/sshd_config)? Did you assign your production webservers an address on interface for the private network as well as on the interface for the public network?  Posting the relevant bits from the ezjail configuration file for one of the jails could be useful.


----------



## junovitch@ (Mar 17, 2015)

At a minimum you have a snapshot of some of your previous configurations from the post.  Have you checked for differences against that?  The FIB approach does take some time experimenting to get the right behavior and tcpdump(1) is going to be you friend.  Do you see SSH sessions using the loopback now or do you see any of them using your external interfaces?


----------



## perkypork (Mar 18, 2015)

asteriskRoss said:


> perkypork, which address(es) did you configure your SSH servers to listen on (in /etc/ssh/sshd_config)? Did you assign your production webservers an address on interface for the private network as well as on the interface for the public network?  Posting the relevant bits from the ezjail configuration file for one of the jails could be useful.



I am running 2 ssh servers, one on base and another with a public IP. Both are configured to only listen on the IP I have allocated them.

I only allocated either a local or public IP (never both) to any one jail.


----------



## perkypork (Mar 18, 2015)

junovitch said:


> At a minimum you have a snapshot of some of your previous configurations from the post.  Have you checked for differences against that?  The FIB approach does take some time experimenting to get the right behavior and tcpdump(1) is going to be you friend.  Do you see SSH sessions using the loopback now or do you see any of them using your external interfaces?



That is a good point. I have also documented what I have done and I guess I will find where I went wrong (or right) by going through that again.

I noticed that with the FIB setup, if I do not have all networking and firewalls for a jail running on Base before I start the jail, it can take some time for the jail to have network access (approx 5-10 minutes). Its sort of like things are running slowly, though my box has 92GB RAM and 12 cores. This issue is not a show stopper, just something to remember.

I have yet to master TCPdump, though I am working on understanding it.

I see no ssh sessions on loopback, not a single thing shows while running `tcpdump -i lo0 -n port 22`

Everything seems to be working great at the moment. My only remaining issue is running two Postgres 9.4.1 jails. I get the "semctl failed: Invalid argument" error message. I will start another post about this as its off topic.


----------



## perkypork (Mar 19, 2015)

As a further follow up, I can say with as much certainty that is possible for running my Jail setup for a few days, that it works, flawlessly.

I am currently running 12 jails, all running different software configurations and different firewall requirements. All of these jails are used to provide a single software platform i.e. they provide different functions for a single software platform). These jails used to be all virtual machines and administering them was extreme painful and time consuming. It used to take approx 1 day to bring up 2 servers, it now takes approx 1 day to bring up 12 servers (jails). In the future, with a little scripting, it should take minutes to bring up these 12 servers (jails).


----------

