# Jails configuration questions



## perkypork (Feb 14, 2015)

We finished moving all our servers over to FreeBSD a couple of weeks ago (we really love FreeBSD) and would now like to leverage Jails to reduce our administration overhead if it fits in with what we are trying to achieve. We currently use Xen for virtualization but the overhead of managing an operating system for each part of our application consumes a heap of time not to mention the waste of server resources. 7 of our servers run nginx and one runs a database.

Our application requires eight different servers, which are duplicated and load balanced to deal with the load. What I would like to achieve is to have these 8 servers running in a jail on a single piece of hardware. My test machine is an Dell R710 with 96GB of RAM, 2 x 6 core processors and 4 network interfaces.

I have come up with the following design:

1 x network interface for management of the base system (probably LAN).
3 x network interfaces for WAN connectivity. Different jails will be allocated different interfaces based on their expected traffic profile.
Our operating system of choice is FreeBSD 10.1

For my first tests I used Ezjail but I don't really mind what I use to achieve what we would like to. I used Ezjail because having flavours makes things much easier to replicate (and we will be replicating each server multiple times). I also like it because it mounts ports via nullfs and I would like to utilise nullfs mounting for each servers nginx files.

What I would like to achieve in essence is 8 separate servers just doing their thing, similar to having 8 virtualised instances of FreeBSD on Xen but without the huge admin overhead.

Hopefully the above gives enough background on what I would like to achieve. Now onto my questions.

When I create a jail with Ezjail and go into its console, running ifconfig shows all interfaces, even if they have not been allocated to the jail. Is this expected behavior or have I misconfigured something?

I have configured the base system to have a different WAN IP (in the same subnet) on each of the interfaces. I have then configured 3 jails which each using a different IP/interface. At this point I like to understand best practice for routing/firewall configuration. Is best practice to allow all traffic on each Jail interface on the base system (being careful to change listen addresses on the base system) and then configure IPFW on each of the Jails or something else?

Should I be using Ezjail or creating each jail the traditional way? I get the feeling that Ezjail may be restrictive in the future but I like many of its features.

I noticed many of the tutorials on the web use PF instead of IPFW. Are there features available in PF that make Jails easier to administer?

If I use IPs on different subnets I assume I will need to use setfib on the base system. Are there any gotcha's in regards to doing this?

Thanks in advance for any help!


----------



## woodsb02 (Feb 14, 2015)

Just answering the parts that I can:



perkypork said:


> When I create a jail with Ezjail and go into its console, running ifconfig shows all interfaces, even if they have not been allocated to the jail. Is this expected behavior or have I misconfigured something?


Yes, this is the expected behaviour. All interfaces will be shown, but only the IP addresses allocated to the jail will show against each interface.



perkypork said:


> Should I be using Ezjail or creating each jail the traditional way?


Obviously it is your preference, but a lot of people (including me) use ezjail because of the convenience it brings, and also the shared basejail minimising the disk space and numerous base system upgrades that would otherwise be required to have many jails.



perkypork said:


> I noticed many of the tutorials on the web use PF instead of IPFW. Are there features available in PF that make Jails easier to administer?


PF is a very popular and capable firewall. It may just be that it was the preference of those writing the tutorials. IPFW is also popular and capable. I'm not sure of any particular features of either that would make one work better with jails than the other.


----------



## gkontos (Feb 14, 2015)

perkypork said:


> If I use IPs on different subnets I assume I will need to use setfib on the base system. Are there any gotcha's in regards to doing this?



No, you don't need to. All your interfaces will be static, so the routing table will be aware of the subnets.


----------



## perkypork (Feb 14, 2015)

gkontos said:


> No, you don't need to. All your interfaces will be static, so the routing table will be aware of the subnets.



Would you mind explaining this further as my testing resulted it not working?

I tested using the following scenario:

2 x interfaces, each with different IPs on different subnets. If I hit IP1 (from any IP) with the defaultrouter on its subnet it responds fine, if I hit IP2 (from a different subnet as IP2) without the defaultrouter on its subnet I get no response at all. Both IPs are static.

defaultrouter = 192.168.0.254

IP1 = 192.168.0.1 netmask 255.255.255.0

IP2 = 10.0.0.1 netmask 255.255.255.0


----------



## perkypork (Feb 14, 2015)

woodsb02 said:


> Yes, this is the expected behaviour. All interfaces will be shown, but only the IP addresses allocated to the jail will show against each interface.



Is there a way to have the jails only show the network interface they have been allocated?



woodsb02 said:


> PF is a very popular and capable firewall. It may just be that it was the preference of those writing the tutorials. IPFW is also popular and capable. I'm not sure of any particular features of either that would make one work better with jails than the other.



I use IPFW because I was under the impression that PF was obsolete on FreeBSD?
https://forums.freebsd.org/threads/ipfw-or-pf.46706/


----------



## wblock@ (Feb 14, 2015)

perkypork said:


> I use IPFW because I was under the impression that PF was obsolete on FreeBSD?



It is not, it has just diverged from the version in OpenBSD.


----------



## kpa (Feb 14, 2015)

perkypork said:


> Is there a way to have the jails only show the network interface they have been allocated?
> 
> 
> 
> ...



IPFW is more capable than PF in some ways, it can do for example Layer2 filtering and PF can not. However, PF has superior rule formalism compared to the very clumsy system used by IPFW that forces you to use GOTO-like constructs to implement even a simple stateful outbound NAT. To say that the version of PF in FreeBSD is obsolete is very far from the truth, it probably comes from the fact that the syntax used in the FreeBSD version of PF is now considered obsolete in OpenBSD and people are mixing up things. Both implementations are roughly equally capable, the only area where OpenBSD has a clear lead at the moment is traffic shaping. The other area where OpenBSD is slightly better is IPv6 where the FreeBSD implementation still can not handle IPv6 fragments and there are some problems with doing NAT with IPv6 traffic (yes it has to be done sometimes).


----------



## gkontos (Feb 15, 2015)

perkypork said:


> Would you mind explaining this further as my testing resulted it not working?
> 
> I tested using the following scenario:
> 
> ...



Lets clear the confusion here. It might be my fault because I did not explain this well.

The defaultrouter statement will reside in the host system and not in any jail. Assuming we have the following scenario:

Host system:

IP: 192.168.0.1/24, bind to interfaceX, default route 192.168.0.254

Jail:

IP: 10.0.0.1/32, bind to interfaceX+1

Then all you need to specify is, gateway_enable="YES", in the rc.conf of the host system.

However, in order to be able to reach 10.0.0.1/32 from a system in the 192.168.0.1/24 lan, you need to specify a static route on the *default router* as well. Example:

`# route add -net 10.0.0.1/24 192.168.0.1`


----------

