# Multiple jails with unique IP addresses



## Elliott (Apr 20, 2020)

I'm interested in setting up a few jails on a small home server.  From what I understand, there are two ways to go about this.

Assign each jail a unique IP address on the same subnet as my home network.  Use port forwarding on my router to route traffic to each one.
Assign each jail a unique IP on a private subnet.  Use `pf` to route traffic internally.

What are the pros and cons of these options?  My gut tells me to use the first option because 2 NATs would add latency and maybe increase CPU / power consumption a tiny bit.  Currently I am not running `pf` at all.


----------



## Elliott (Apr 20, 2020)

On a similar note, what is the purpose of assigning two IP addresses to one jail?  I followed the instructions here:


			14.6. Managing Jails with ezjail
		


It says the purpose is 





> "To keep jail loopback traffic off the host's loopback network interface lo0"


But then it also says





> "Inside a jail, access to the loopback address 127.0.0.1 is redirected to the first IP address assigned to the jail"


The example command is 

`ezjail-admin create dnsjail 'lo1|127.0.1.1,em0|192.168.1.50'`


Why not just assign one IP address 192.168.1.50 ?  Would this not achieve the same thing?  This is actually causing a problem for me because Plex Media Server binds to the first interface.


----------



## zirias@ (Apr 20, 2020)

A "traditional" jail shares the network stack with the host and all other jails. This means the only separation happens on the "IP" level, you can restrict a jail to specific IP addresses. As all of these addresses are created on interfaces of the host system, the host system sees them all, and you have to be careful to configure any network services to specifically listen on the IP addresses you want.

That said, I think you misunderstood something when you write "2 NATs". If you want to expose a service in a jail to the outside via a public IP address, you need to do NAT exactly once, no matter whether your jails are all sharing a subnet or have their own. The difference is that with separate subnets, you are in control of the routing _between_ jails (and the host), so you could apply firewall rules there as well.

For some time now, you can go one step further and use VNET jails, which means the jail has its own private network stack -- you typically give it an epair(4) interface that's completely separate from the host's interfaces. The other end of the epair can be routed (like giving each jail its own subnet) or bridged (somewhat similar to have all jails share a common subnet). IMHO, this concept is easier to grasp and gives an even better separation, so I personally recommend using VNET jails.


----------



## rootbert (Apr 20, 2020)

Zirias said:


> IMHO, this concept is easier to grasp and gives an even better separation, so I personally recommend using VNET jails.


+1 recommendation for VNET jails from my side!


----------



## Elliott (Apr 20, 2020)

I'm not sure if ezjail can do VNETs.  It sounds like I need to abandon that and read up on `jails.conf`.

Regarding the 2 NATs, my home has one public IP address, NATted by the router to 192.168.1.0/24.  In my first example using pf, each jail is in the 127.0.1.0/24 subnet, so I would access the server at 192.168.1.5 and this would translate to 127.0.1.1.  Isn't that considered double NAT?


----------



## sand_man (Apr 21, 2020)

Right, I don't believe ezjail uses VNET but pretty sure iocage does.


----------



## Hakaba (Apr 21, 2020)

I recommand learning / experimenting jails without ezjail/iocage/bastille... 
Because once you know what FreeBSD jails is, you understood all this tools.
A manager tool is useless before the «industrialization» step.


----------



## rootbert (Apr 21, 2020)

Hakaba said:


> I recommand learning / experimenting jails without ezjail/iocage/bastille...
> Because once you know what FreeBSD jails is, you understood all this tools.
> A manager tool is useless before the «industrialization» step.


+ 1 for that ... managing jails with the tools the base system is giving you is very straight forward! In addition to that, iocage cost me hours for its bugs (I call it bugs, the author calls it design decisions), my tipp: stay away from iocage especially if you plan on using VNET jails, go with the systems standard tools!


----------



## Elliott (Apr 21, 2020)

Yes, I generally prefer to do things manually for better understanding, at least to start with.

I'm trying to start simple, and so far I have not had any need for downloading source code.  I'm installing packages, not ports.  But all the documentation I read about jails uses something like this

`D=/here/is/the/jail
     cd    /usr/src
     mkdir -p $D
     make world    DESTDIR=$D
     make distribution DESTDIR=$D`

Is it required to build the jail from source or is there a faster way?


----------



## suntzu00 (Apr 22, 2020)

just download the sources(extract them, etc) and you're good to go

https://www.freebsd.org/doc/handbook/jails-build.html


----------



## ccammack (Apr 24, 2020)

My use case is similar to yours, and `iocage` works well for me.

It looks like there is a Plex Media Server plugin for `iocage`, which should simplify configuration if you decide to try it. I have never used it.

I see things the other way; use iocage to get a test jail working, then examine the files it produces to understand what the underlying system is doing.


----------



## neel (Apr 25, 2020)

To answer OP's question, Option 1 (what I do with `iocage`) means all ports are accessible within your home network. This is simpler as there is less moving components, but every port on your jails is accessible to your whole network unless you firewall on the jail itself.

Option 2 (used by Docker) saves IPv4 addresses. This is useful on cloud providers with 1 public IP/VM but not as much your home LAN which is already behind NAT. This has the disadvantage of more complexity (`pf`/`ipfw` needed).

If you use Jails on a home network, I'd recommend Option 1. Less moving components and it's not like your LAN is running out of internal addresses. And NAT is heavier than just pure firewalling if you need it.


----------



## petlib (May 1, 2020)

Elliott said:


> Yes, I generally prefer to do things manually for better understanding, at least to start with.
> 
> I'm trying to start simple, and so far I have not had any need for downloading source code.  I'm installing packages, not ports.  But all the documentation I read about jails uses something like this
> 
> ...



You can do this:
`mkdir /jails/new_jail
bsdinstall jail /jails/new_jail
freebsd-update -b /jails/new_jail fetch install`


----------



## octix (May 1, 2020)

I tried initially with pure jails, but gave up for the time being, too many unknowns and not enough time. Eventually got it working with `iocage` + `vnet`. I have not used it that much to get into any bugs or major issues. I like the concept of releases (don't have to manually create the first jail) and templates. Easy to work and automate (for me at least). I have a base R12.1 template with my ssh key, which I clone and assign a separate IP. After that I use ansible to finish setup.

I could not figure out a simple way to automate updating /etc/jail.conf with new jails, because it's a single file. I wish there were includes and/or separate files.

This is my script to create new jails

```
#!/bin/sh

JAIL_TEMPLATE=$1
JAIL_NAME=$2
JAIL_IP=$3

iocage create -T -t $JAIL_TEMPLATE -n $JAIL_NAME
iocage set allow_raw_sockets=1 vnet=on ip6_addr=none defaultrouter6=none defaultrouter=192.168.1.1 ip4_addr="vnet0|$JAIL_IP/24" $JAIL_NAME
iocage start $JAIL_NAME
```

Btw, this is how `iocage` uses `freebsd-update` to upgrade jails (saw that while having `htop` running).

```
/bin/sh /tmp/tmp3cn1hmz9 -b /z/iocage/jails/test1/root -d /z/iocage/jails/test1/root/var/db/freebsd-update/ -f /z/iocage/jails/test1/root/etc/freebsd-update.conf --not-running-from-cron fetch
```

I had an issue with `iocage pkg` though, it was either super slow or froze, but it's not a blocker I can use ansible and pkgng


----------

