# What's the best place to find jail management instructions?



## Toolforger (Mar 2, 2020)

Hi all,

TL;DR: I have a single FreeBSD host with a few datasets, each dataset has a few jails, what's the best way to set up and manage those jails and where to I get the docs to do that? (Each dataset is one protection domain.)
I believe that ezjail is the way to go, I believe that I don't need VIMAGE but I'm not sure, I don't know if there are additional features I should check.

Details:

I've been checking handbook and various tips&tricks pages about jails, and things are getting more confusing by the day - there are just too many details, many pages seem to be outdated (but not mention the FreeBSD version, grr), and I don't want to spend days on an approach, just to find that that it was outdated and possibly misguided. And I'm seeing references to VIMAGE and don't know what the status is, or if it is useful - probably no but who knows? And I don't know if there are more such features that may or may not be applicable, and it's hard to google for things that you don't even know the name of, or if they exist at all.

So... what's the most useful set of instructions for organizing jails?

I am aware that the answer may depend on the use case, so here is mine, just to give you an idea of the level of (non-)complexity I need to handle:

1. The jails run inside a single FreeBSD host (i.e. no clustering or failover, just good old ZFS backup to a separate machine).

2. There are multiple ZFS datasets (two for now).
Each dataset is operated upon by a couple of jails (one to retrieve data from the LAN; one to offer a read-only view of the data (sambad or rsyncd); maybe a webserver to display statistics and such).
There is currently no use case that requires that a jail accesses multiple datasets.

3. There are no end users on the FreeBSD host.
(I will likely assign a separate user id for each dataset, as an extra layer of protection. I don't know how much that will buy me.)

4. If there's a trade-off between reliability and something else, I will always favor reliability.


----------



## rigoletto@ (Mar 2, 2020)

The easiest way to manage Jails is using management tool and there are plenty: sysutils/iocage[1], sysutils/bastille, sysutils/cbsd etc.

[1] I would recommend the -devel version, I had some issue (I don't remember now) which is fixed on -devel.


----------



## SKull (Mar 2, 2020)

I too recommend iocage.
I don't recall problems with the non devel version. But my only use case is service isolation anyway.


----------



## Emrion (Mar 2, 2020)

rigoletto@ said:


> The easiest way to manage Jails is using management tool and there are plenty: sysutils/iocage[1], sysutils/bastille, sysutils/cbsd etc.
> 
> [1] I would recommend the -devel version, I had some issue (I don't remember now) which is fixed on -devel.


sysutils/iocage is (was?) serioulsy bugged if you apply an update with `iocage update`. It somehow loose its devfs_ruleset and the jail doesn't restart until you fix this number at what it should be (4 in general).

I updated sysutils/iocage yesterday (by `pkg` on LATEST repo) and don't know whether if this bug persists or not. But there is another annoying bug which makes iocage unworkable if LANG is set to something else than xx_xx.UFT-8. I found how to fix it (just added one line of code), made a bug report but nothing was corrected in this new version.

I will try the devel version if possible. I was looking at sysutils/iocell but it seems complicated_ at least_ to keep jails created by iocage. That being said, iocell has none of the two bugs I mentioned concerning iocage.


----------



## Bobi B. (Mar 2, 2020)

Take a look at FreeBSD Mastery: Jails book by Michael W Lucas.


----------



## SKull (Mar 2, 2020)

Bobi B. said:


> Take a look at FreeBSD Mastery: Jails book by Michael W Lucas.


Which recommends using iocage 
Great book nonetheless!


----------



## rigoletto@ (Mar 2, 2020)

Emrion said:


> sysutils/iocage is (was?) serioulsy bugged if you apply an update with `iocage update`. It somehow loose its devfs_ruleset and the jail doesn't restart until you fix this number at what it should be (4 in general).
> 
> I updated sysutils/iocage yesterday (by `pkg` on LATEST repo) and don't know whether if this bug persists or not. But there is another annoying bug which makes iocage unworkable if LANG is set to something else than xx_xx.UFT-8. I found how to fix it (just added one line of code), made a bug report but nothing was corrected in this new version.
> 
> I will try the devel version if possible. I was looking at sysutils/iocell but it seems complicated_ at least_ to keep jails created by iocage. That being said, iocell has none of the two bugs I mentioned concerning iocage.



Until recently (at least) you couldn't update the jails at all with sysutils/iocell because FreeBSD was made some change in the files (removed one) and sysutils/iocell can't fetch it and then fail. Thus, I would say sysutils/iocell is Abandonware now.


----------



## Emrion (Mar 2, 2020)

rigoletto@ said:


> Until recently (at least) you couldn't update the jails at all with sysutils/iocell because FreeBSD was made some change in the files (removed one) and sysutils/iocell can't fetch it and then fail. Thus, I would say sysutils/iocell is Abandonware now.


Good to know.

After some more tests, I found that the last version of iocage is running ok for updating jails. And also I can use iocage created jails with the devel version. Furthermore the iocage-devel has none of the bugs I was speaking of, not even the "LANG" one. Thanks.


----------



## Toolforger (Mar 2, 2020)

Thanks to all.
Seeing that nobody recommends ezjail - does ezjail have issues?
I mean - the iocage reports sound like it's somewhat flaky and near the borderline to abandonware, with no idea on what side of the border it is - if that's the best recommendation, paint me worried.


----------



## rigoletto@ (Mar 2, 2020)

Toolforger said:


> Thanks to all.
> Seeing that nobody recommends ezjail - does ezjail have issues?
> I mean - the iocage reports sound like it's somewhat flaky and near the borderline to abandonware, with no idea on what side of the border it is - if that's the best recommendation, paint me worried.



I think sysutils/ezjail is abandonware now. Welcome to the OSS world!

sysutils/iocage-devel is being developed, had an update recently but just not a new release yet; however I would give a shot to Bastille if I had not several jails yet.


----------



## m0nkey_ (Mar 2, 2020)

I went the other way from `iocage` to just editing my own jail.conf and pulling down base. Iocage broke on my more than enough times to give up on. Now I just keep snapshots of a base jail, clone it then edit /etc/jail.conf. Hasn't broken yet and no more difficult than using iocage.


----------



## Toolforger (Mar 3, 2020)

rigoletto@ said:


> The easiest way to manage Jails is using management tool and there are plenty: sysutils/iocage, sysutils/bastille, sysutils/cbsd etc.



Bastille looks nice, but it seems to insist that you manually assign an IP address to each jail.
I'd really prefer if it allowed me to use DHCP/SLAAC for that. I could manage the addresses manually, but anything that you need to manage once every two years means that you either have to re-read all the details, or overlook some crucial detail and get held up.


----------



## Toolforger (Mar 3, 2020)

rigoletto@ said:


> The easiest way to manage Jails is using management tool and there are plenty: sysutils/iocage, sysutils/bastille, sysutils/cbsd etc.



cbsd seems to do far too many things, including web management consoles.

This is certainly useful, but not for my use case: My FreeBSD box is supposed to be the trusted management anchor for the LAN, so it doesn't have sshd or anything else that allows anybody to manage it via the network. Just a console and a keyboard.
cbsd also does a lot of stuff I don't need (bhyve support, network consoles, and such), and since I want to keep the attack surface small, cbsd isn't usable.

With ezjail abandoned, iocage not really well-updated, bastille almost-good and cbsd not fitting - are there other candidates in the "etc." category worth consideration?


----------



## Toolforger (Mar 6, 2020)

I guess I'll try the manual approach after reading the Jails book.
Thanks for all the feedback!


----------



## zomar (Mar 6, 2020)

I tried few management tools. But for full control on jails I prefer "Plain vanilla management". Take a look at: https://genneko.github.io/playing-with-bsd/system/learning-notes-on-jails/ 

Managment tools for me are ok for testing. But for production I need as small as posible chain of depenency. Upgrading jails in long term will be less stresfull.

If you will play longer with vanilla jails try to write your own SH scripts for common tasks.


----------



## michael_hackson (Mar 6, 2020)

I always go by the guide from scottro.

https://www.scottro.net/nullfsjail.html

That is the best hands on guide I have found, it has made jail management so much easier.


----------



## zomar (Mar 6, 2020)

michael_hackson said:


> I always go by the guide from scottro.
> 
> https://www.scottro.net/nullfsjail.html
> 
> That is the best hands on guide I have found, it has made jail management so much easier.



This tutorial is outdated. Also I see mounting fs from host to jail as a kind of dependency. I like THICK jails. Disk space is cheap nowdays. They are more portable between hosts. 

Mounting fs from host can be partly avoided by using for example nginx cache proxy in another jail as source for pkg, ports, jail update/upgrade process.

Tutorial which I linked in my previous post is no ideal, but is a good and actual base for vanilla jail managemet.


----------



## zomar (Mar 6, 2020)

Toolforger said:


> 2. There are multiple ZFS datasets (two for now).
> Each dataset is operated upon by a couple of jails (one to retrieve data from the LAN; one to offer a read-only view of the data (sambad or rsyncd); maybe a webserver to display statistics and such).
> There is currently no use case that requires that a jail accesses multiple datasets.



In my opinion each jail should have separate zfs dataset.

Jail can have more mounted datasets. For example:
1) Jail with mysql should have extra one or more datasets for db files (according to http://open-zfs.org/wiki/Performance_tuning#MySQL)

2) jail for ZFS replication backup can have a lot datasets. Such jail have to be prepared to manage of the dataset from within a jail (see Jails section of https://www.freebsd.org/cgi/man.cgi?zfs(8) ).


----------



## Toolforger (Mar 6, 2020)

zomar said:


> In my opinion each jail should have separate zfs dataset.
> 
> Jail can have more mounted datasets. For example:
> 1) Jail with mysql should have extra one or more datasets for db files (according to http://open-zfs.org/wiki/Performance_tuning#MySQL)
> ...



Let me rephrase the core point I was trying to make:

I'd agree in general, but I want the one writer process in a separate jail than the reader processes, for security reasons.

Sure, that fits for the general case.

However, I want the system to be designed so that it is possible to identify all scripts that may write to a dataset, while retaining the ability to add reader scripts without having to worry about mistakes.
Having the dataset mounted r/w for one and only one jail should nicely achieve that. Plus, I could add reader jails at will, experiment with different tools or different tool versions, without any possibility that I may impact the writer process. (I could probably do similar by mounting the same dataset into a separate mountpoint for each process. This wouldn't allow me to play with the software I install at the "system" level inside each jail, so I believe jails would still be the best option.)


----------



## zomar (Mar 9, 2020)

Toolforger said:


> Let me rephrase the core point I was trying to make:
> 
> I'd agree in general, but I want the one writer process in a separate jail than the reader processes, for security reasons.



I do not understand your goals. 

General tips:
1) Do not try to over complicating zfs/jail configuration. Do not push technology to its limits. Think about that there will be a day when you will need to recreate envirioment. Especialy if you are going to use third party jail managemt tools.

2) Mayby there are other technologies, services or packages to achieve goal. If yes think about zfs/jail as a virtualization platform and all adventages of it.

Good luck!


----------

