# [Guide] A 'Shell take' on Jails



## ShelLuser (Mar 25, 2018)

*Editorial*

It has been two times now that relying on a Jail managed to get my system and myself out of a heap of trouble. The first time was when I was unable to build a certain port but the error messages made me suspicious; I didn't believe that this could have been a screw up from the ports maintainer. Then it hit me: my jail! I rebuild the port within the (clean!) environment of my jail (using the same configuration, obviously) and it build cleanly. Examining the logs helped me to find the cause of the problems (a dumb experiment a few months prior) and fix it.

A continuance of this is my base system itself. Yesterday I tried to rebuild my kernel only to be greeted with an error message. As it turned out this was caused by a (self inflicted!) problem with Clang. And once more it was my jail which helped me to fix it by building the entire base (and kernel) within the confines of said jail.

Time for me to give a little love back and share this with you guys. I'm well aware that most (?) of you will probably be well familiar with the whole concept already, but these writeups serve more purposes   And there's always a chance that some people might actually learn something new here.

This will not be your standard "how to install a Jail" guide   Sorta...

*What is a Jail?*
A Jail is basically a virtual FreeBSD environment running "inside" your main environment. Somewhat comparable to what you can achieve with chroot, and since I like to go all out in my guides lets start by taking a closer look at this mysterious chroot command.

*Change the root ("chroot")*

As you probably know the heart of a Unix system (other than the kernel and/or init) is the root. It all starts at / which is the 'beginning' of your system; the whole directory hierarchy starts here. Unix doesn't know about drive letters and all that nonsense * but instead places everything within "the one" directory structure. But this does pose a risk. For example: /dev/mem basically provides access to your system's memory. Or /etc/fstab which could help an attacker to learn more about your setup.

But there are also less paranoid reasons: if a user has all their data in their home directory, then why bother them with stuff they can't use in the first place? Why not confine them in a place of their own? This is what chroot does. Once again: the root is the heart of your Unix(-like) system, and this allows you to "shift" that perception onto another location within your Unix environment.

However, it is not a command you can "just" use. It needs some preparations:


```
peter@unicron:/home/peter $ ls       
Desktop/                devel/                  mc/
Mail/                   downloads/              passwd
NetBeansProjects/       etc/                    pictures/
VisualParadigm/         infocom/                spigot@
backups/                kamps/                  temp/
bin/                    mail/                   transfer/
bugzilla.png            man/
peter@unicron:/home/peter $ chroot temp
chroot: temp: Operation not permitted
peter@unicron:/home/peter $ su
Password:
root@unicron:/home/peter # chroot temp
chroot: /bin/csh: No such file or directory
root@unicron:/home/peter # ls /bin/csh
/bin/csh*
```
Pay close attention to that last error. As you can see the error seems bogus because when I use `ls /bin/csh` it shows up easily. So what's this about "No such file"? Well... this is what chroot is all about. /bin/csh exists. But the chroot command _changes_ the root to a new location, in my attempt above I tried to change this to /home/peter/temp. So we need to exchange / for /home/peter/temp and guess what?


```
root@unicron:/home/peter # file /home/peter/temp/bin/csh
/home/peter/temp/bin/csh: cannot open `/home/peter/temp/bin/csh' (No such file or directory)
```
That's what's triggered our error message. But no problem: we'll just copy the executable and be done with it. Well... maybe not:


```
root@unicron:/home/peter # mkdir temp/bin
root@unicron:/home/peter # cp /bin/csh temp/bin
root@unicron:/home/peter # chroot temp
ELF interpreter /libexec/ld-elf.so.1 not found, error 2
Abort
```
Have you ever wondered what the big deal with /rescue is? If so: here you go. See: executables within FreeBSD rely on libraries to function. Sometimes quite a few of them. If you want to learn more about that then ldd can be a very useful tool:

```
root@unicron:/home/peter # ldd `which csh`
/bin/csh:
        libncursesw.so.8 => /lib/libncursesw.so.8 (0x800885000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x800ae3000)
        libc.so.7 => /lib/libc.so.7 (0x800d02000)
```
So in order to make this work we'd have to copy all those libraries within our new root environment before csh can run. _...or_. Did you know that all the files within the /rescue location have been "statically compiled"? What that means? _This:_

```
root@unicron:/home/peter # ls -l /bin/csh /rescue/csh
-r-xr-xr-x    2 root  wheel    432272 Mar 25 11:26 /bin/csh*
-r-xr-xr-x  139 root  wheel  10901936 Mar 25 11:27 /rescue/csh*
```
Notice something? /rescue/csh is a lot bigger than the other executable. The reason for that is that /rescue/csh does not rely on any other external libraries. They're combined into one package. In official terms: "/rescue/csh is not a dynamic ELF executable". Instead it's self-supporting. As such:

```
root@unicron:/home/peter # cp /rescue/csh temp/bin
root@unicron:/home/peter # chroot temp
# ls /
ls: Command not found.
# cd /usr/local
/usr/local: No such file or directory.
```
Please don't tell me that you're not getting a little bit excited here? 

So what is happening? I'm logged in as root and the default shell for that user is /bin/csh. I'm also trying to change my root ("chroot") to /home/peter/temp. By doing so I'm basically telling the system "please execute /bin/csh and confine it to /home/peter/temp". The result can be seen above. Since I never copied /bin/ls to /home/peter/temp/bin the command cannot be started. Same for /usr/local.

I basically told the system to exchange / for /home/peter/temp. So what was once / has now physically become /home/peter/temp. Pretty cool, right?

*So what about Jails?*

A jail takes the above concept one step further. Instead of merely locking a process into a specific file system location it will actually "fork" a full blown kernel process and start that within the confines of a specific location. In other words: it will start a full blown - yet virtual - FreeBSD environment and limit its access not only to that specific filesystem location. No, this goes much deeper than that: because this is a separate kernel process it will _also_ generate a whole new "userland environment". In other words: it separates kernel processes. If you want to know what processes are running then you can use ps for that info. ps will then ask the kernel and relay that info back to you. But by starting a new kernel process ps would ask the kernel about this while that kernel process in itself has absolutely no knowledge about anything happening outside its own perimeter. As a result anything inside the jail would be oblivious to everything else.

Talk is cheap so lets put this to the test:

```
root@unicron:/home/peter # mount -t devfs devfs temp/dev
root@unicron:/home/peter # chroot temp
# ps
  PID TT  STAT    TIME COMMAND
1145 v0  IWs  0:00.00 login [pam] (login)
61750 v0  S    5:30.32 /usr/local/bin/X :0 -auth /home/peter/.serverauth.61736 (  1146 v1  Is+  0:00.01 /usr/libexec/getty Pc ttyv1                                1147 v2  Is+  0:00.01 /usr/libexec/getty Pc ttyv2                                1148 v3  Is+  0:00.00 /usr/libexec/getty Pc ttyv3                                1149 v4  Is+  0:00.01 /usr/libexec/getty Pc ttyv4                                1150 v5  Is+  0:00.01 /usr/libexec/getty Pc ttyv5                             
1151 v6  Is+  0:00.01 /usr/libexec/getty Pc ttyv6
2778  4  IW   0:00.00 su
2779  4  IW   0:00.00 _su (csh)
2782  4  I+   0:01.48 mc
2783  5  Is+  0:00.38 /bin/csh
64628  7  IW   0:00.00 su
64629  7  S    0:00.09 _su (csh)
64886  7  S    0:00.01 /bin/csh -i
64889  7  R+   0:00.00 ps
```
Note: my server restricts process listings to current users only. Even so, now lets try the same thing but this time from within a jail:

```
root@unicron:/home/peter # jexec psi ps
  PID TT  STAT    TIME COMMAND
64955  7  R+J  0:00.02 ps
```
That's quite the difference, wouldn't you agree? All the system noticed was the same process we just started.

This is what sets jails apart from a chroot'ed environment: full separation. As mentioned earlier it's basically a virtual system running on top of your current one.

*How to set it up?*

There are tons of ways to do this and there's honestly no "best" here. Never forget: what works for me doesn't have to work for you. But having said that I do think I have some useful tips to share...

Now, before I start I need to mention sysutils/ezjail. Note that I have 0 experience with this port but I know plenty of people use this to manage their ports so it seems only fair to mention it. If you want easy then this could be for you.

Thing is: I don't want easy because 'easy' always comes with a price attached. Even here. For starters: uniformity. If you use this port then I'm pretty sure you'll confine yourself to its set out standards. Standards which are out in the open for everyone to learn. So once someone finds out their inside a jail and they're familiar with ezjail it just might give them an edge if they're also gained access to your host. Unlikely, for sure, but still something to consider. Especially when security is an issue.

Another aspect is that setting up a jail really isn't all that hard.

*Userland*

As mentioned above a jail is basically a separate kernel process, and what does a kernel need? A userland! So basically an environment to use and allow usage from users. It is for that reason why I prefer to set up a jail environment with a copy of the general base system. In other words:

```
peter@unicron:/home/peter $ zfs list | grep jails
zroot/opt/jails        6.20G  97.0M   153M  /opt/jails
zroot/opt/jails/psi    6.05G  49.4G  6.05G  /opt/jails/psi
peter@unicron:/home/peter $ ls /opt/jails
base.txz        kernel.txz      lib32.txz       psi/
```
My server runs FreeBSD 11.1 and what you see above are the base packages for that OS. My psi jail is basically nothing more but an extracted copy of base.txz (and a bit of manual tweaking).

This is the first step in setting up a jail: the userland.

I prefer to set up a directory (actually: a virtual ZFS filesystem) and then extract the base.txz in there so that I have a pristine FreeBSD setup. Next stop: configure the jail, that can be done using /etc/jail.conf:

```
peter@unicron:/home/peter $ cat /etc/jail.conf
## Jail(s) config

psi {
        host.hostname = "psi.intranet.lan";
        ip4.addr = 10.0.1.6;
        path = /opt/jails/psi;
        mount.devfs;
        mount.fstab = /etc/fstab.psi;
        persist;
        interface = em0;

#       devfs_ruleset = 2;
#       enforce_statfs = 1;

        allow.raw_sockets;
        allow.chflags;
        allow.mount.procfs;
        allow.mount.devfs;
}
```
This is pretty much all you need. A hostname, valid IP address and the path to your set up userspace followed by some optional preferences. For this jail I took things easy and set it up so that the IP address would become an alias on em0 which is my main network interface. But you can easily change this if you want. My setup gives the jail pretty much unlimited network access. A good way to prevent this is to utilize lo0 and then set up a NAT connection between your jail and the main interface.

*Fire it up!*

When everything has been set up all you have to do is actually start, or _create_, your jail. To be honest I think this is a poor choice of wording but at the same time I cannot come up with anything else. I mean... To me the process of creation is done when you set up the base system (for example by extracting base.txz or by installing the userland manually using `# make DESTDIR=/path/to/jail/root installworld`). But in all honesty: this only becomes a real jail once you "forked" the kernel process: `# jail -c psi`.

The cool part here is that because your jail is a basically a virtual environment you can even associate processes to be used within the jail. In normal English: set up /etc/rc.conf in order to automatically start certain processes. So that whenever you start the jail you'll also (automatically) start those processes. Just like you'd do on a "normal" FreeBSD system.

Which brings me to my last item... How do we start all this?

Well, using /etc/rc.conf:

```
jail_enable="YES"
jail_list="psi"
```
This code snippet ensures that whenever my server boots it'll also fire up my Psi jail.

*Why would you want a jail?*

There are many purposes. Security (basically: an extra security _layer_) is one of those. Install any public services into a jail and if one gets run over due to some freak backdoor then the attacker will only be confined within your jail, so mostly ruling out any risk for the base system. Reset the jail, and you're good to go again.

But there's more. Are you building your own ports? I do too. The problem though is that a lot of this depends on your environment. So what if you have some kind of problem with said environment (maybe a library which you forgot to rebuild against other (upgraded) libraries)? That's when a jail can become quite useful. Because it's separated from your main host it wouldn't be "contaminated" in the same way, thus providing the perfect backup as building environment.

Of course you should always be careful here, because although you may have ruled out the dependency on certain libraries during the build process, what about the actual usage process?

What to think about DMZ? Aka: de-militirazed zones. No, not to host public services, but what about running your next browsing session from within the confines of a jail? Downloaded any malware? Who cares if it was specifically targeted for FreeBSD: you're inside a jail so worst case scenario is that your jail gets messed up. A quick reset and you're good to go again.

_And there you have it...._

A different take on Jails. I'm well aware that most I shared is also documented elsewhere, such as the awesome FreeBSD handbook, but I still hope that this could be useful for some of you. Jails are amazing. Or at least they can be 



Spoiler: *



Ok, a small disclaimer.. I don't really think drive letters are stupid, and I can even understand the motivation behind them. Let's not forget the target audience here: DOS / Windows tries to cater to a larger user base, including people who don't share our expertise. So trying to keep a separation between the system (C:) and an optional data volume (D:)? I honestly think it's actually a good idea!

But...  This is FreeBSD we're talking, we're amongst greybeards here. And within the Unix mindset a drive letter is plain out nonsense; a mountpoint is all we need. As such the comment above.


----------



## reddy (Dec 23, 2018)

I am not sure about user management for jails. I am wondering what people are usually doing:

1. Create users from the host using jexec and essentially interact with the jail through a shell as they would with the host
2. Create jail users in the host, restrict file system access to specific folders within the jail and execute processes in the jail as that user using jexec -U from the host

The specific scenario I have in mind is the one of a www jail. Multiple software development teams developing (and therefore deploying multiple applications), through a FTP service running within the jail. Not every team should have access to every application folder within the www jail, thus the need for  proper user management.


----------



## ShelLuser (Dec 24, 2018)

reddy said:


> 1. Create users from the host using jexec and essentially interact with the jail through a shell as they would with the host
> 2. Create jail users in the host, restrict file system access to specific folders within the jail and execute processes in the jail as that user using jexec -U from the host


It depends on context. For example: if you're using a jail to host processes then you have no other choice but to create local user accounts as well, otherwise the process wouldn't be able to run in the first place (depending on the process of course, and this doesn't account for abusing the root account for it, that would be a horribly bad idea in the first place).

Also note that -U wouldn't work without local (jail) accounts, see also jexec(8). There's also the issue that jexec cannot be used by regular users.

Most common usage is to set up a jail, populate it with software which you want to use and then set up that software within the jails rc.d structure. This will ensure that the service will start as soon as the jail starts, which can be done as soon as the host starts.



reddy said:


> The specific scenario I have in mind is the one of a www jail. Multiple software development teams developing (and therefore deploying multiple applications), through a FTP service running within the jail. Not every team should have access to every application folder within the www jail, thus the need for  proper user management.


I don't see how the use of jexec would even be a viable option here. So just set up the jail, set up the user accounts inside the jail and utilize all that. Anything beyond that would simply overcomplicate things.

Also: I find it much easier to simply run `# jexec psi` (where psi is my jail name) after which I'm logged onto the jails console and can perform all my maintenance from there. Having to use jexec for every single command run on the jail seems like a terrible waste of effort to me.



pyret said:


> Be careful with SysV IPC being shared across jails. If you are running SmartOS and use KVM or bhyve


This is about FreeBSD, not about any derivatives. But mostly: how would you even set this up in the first place considering that every jail is pretty much separated.



> because an attacker would have to find an exploit in them and get out, but then they'd be inside a Solaris Zone, and would have to find an exploit to get out.


This makes no sense what so ever, where did Solaris zones suddenly come into all this?


----------



## twllnbrck (Dec 24, 2018)

Thanks ShelLuser, nice short and informative guide, as ever!


----------

