# FreeBSD 10 jail on nullfs + unionfs + devfs causing pain



## amiramix (May 26, 2014)

Hello all,

I am trying to set up a simple jail following various tutorials and designs for the underlying file system. It works when I do all the steps manually but fails when I try to automate it with jail.conf. In particular it fails to mount devfs in the jail, which makes the jail pretty unusable. The manual steps are like this:


```
zfs create tank1/j/_ro
bsdinstall jail /j/_ro
zfs create tank1/j/_tst
mkdir /j/tst
mount_nullfs -o ro /j/_ro /j/tst
mount_unionfs -o rw,noatime /j/_tst /j/tst
mount -t devfs -o rw,ruleset=4 devfs /j/tst/dev
```

This is essentially installing a read-only base system, then mounting unionfs on top of it to allow changes specific to a jail, then finally adding a devfs for the jail to use. After executing those steps /j/tst/dev contains the proper set of devices, as declared by the ruleset 4.

Now I use this jail.conf script to start the jail:


```
# exec.prestart = "/sbin/mount -F /usr/local/etc/fstab/$name -a";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
# exec.poststop = "/sbin/umount -F /usr/local/etc/fstab/$name -a";
exec.clean;
# mount.devfs;

path = "/j/$name";
host.hostname = "$name.myhost.com";
# mount.fstab = "/usr/local/etc/fstab/$name";

tst {
  ip4.addr = 192.168.0.5;
  interface = em0;
#  devfs_ruleset = 4;
}
```

I use different values for .myhost.com, ip and em0 - they are not important here. But please note the commented out lines. I can now start the jail and the devfs is still properly set up:


```
[root@myhost ~]# service jail start tst
Starting jails: tst.
[root@myhost ~]# ls -l /j/tst/dev/
total 1
dr-xr-xr-x  2 root  wheel      512 May 26 15:36 fd
crw-rw-rw-  1 root  wheel      0x7 May 26 15:36 null
dr-xr-xr-x  2 root  wheel      512 May 26 15:36 pts
crw-rw-rw-  1 root  wheel      0xc May 26 10:25 random
lrwxr-xr-x  1 root  wheel        4 May 26 15:36 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 May 26 15:36 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 May 26 15:36 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 May 26 15:36 urandom -> random
crw-rw-rw-  1 root  wheel      0x8 May 26 10:25 zero
crw-rw-rw-  1 root  operator  0x3a May 26 10:25 zfs

[root@myhost ~]# jexec tst sh
# ls -l /dev/
total 1
dr-xr-xr-x  2 root  wheel      512 May 26 14:21 fd
(...)
crw-rw-rw-  1 root  operator  0x3a May 26 10:25 zfs
```

Now is the tricky part. I shutdown the jail, unmount all devices mounted manually above, and try to mount the filesystem automatically by commenting in either the lines with the exec.prestart/exec.poststop scripts or the mount.devfs/mount.fstab/devs_ruleset configurations in the jail.conf (above). For completeness the fstab looks like this:


```
[root@myhost ~]# cat /usr/local/etc/fstab/tst
/j/_ro  /j/tst     nullfs  ro           0 0
/j/_tst /j/tst     unionfs rw,noatime   0 0
```

In either case the devfs isn't being mounted properly. Command `mount` shows the mounts, but the content of /j/tst/dev is essentially empty. There are just two files probably created automatically when the jail was being started:


```
[root@myhost ~]# service jail start
Starting jails: tst.
[root@myhost ~]# mount
(...)
/j/_ro on /j/tst (nullfs, local, read-only)
<above>:/j/_tst on /j/tst (unionfs, local, noatime)
devfs on /j/tst/dev (devfs, local, multilabel)
[root@myhost ~]# ls -l /j/tst/dev/
total 1
lrwxr-xr-x  1 root  wheel  12 May 26 15:26 log -> /var/run/log
-rw-r--r--  1 root  wheel   0 May 26 15:32 null
```

Note the difference between the content of /j/tst/dev above and after I mounted the devfs manually. I see exactly the same result if I use the prestart/poststop scripts in the jail.conf as described above. It only works (and I discovered this by accident) when I add an additional line to the fstab file:


```
[root@myhost ~]# echo "devfs   /j/tst/dev devfs   rw,ruleset=4 0 0" >> /usr/local/etc/fstab/tst
[root@myhost ~]# cat /usr/local/etc/fstab/tst
/j/_ro  /j/tst     nullfs  ro           0 0
/j/_tst /j/tst     unionfs rw,noatime   0 0
devfs   /j/tst/dev devfs   rw,ruleset=4 0 0
[root@myhost ~]# service jail start
Starting jails: tst.
[root@myhost ~]# mount
(...)
/j/_ro on /j/tst (nullfs, local, read-only)
<above>:/j/_tst on /j/tst (unionfs, local, noatime)
devfs on /j/tst/dev (devfs, local, multilabel)
devfs on /j/tst/dev (devfs, local, multilabel)
[root@myhost ~]# ls -l /j/tst/dev/
total 1
dr-xr-xr-x  2 root  wheel      512 May 26 15:36 fd
(...)
crw-rw-rw-  1 root  operator  0x3a May 26 10:25 zfs
[root@myhost ~]# jexec 33 sh
# ls -l /dev/
total 1
dr-xr-xr-x  2 root  wheel      512 May 26 15:36 fd
(...)
crw-rw-rw-  1 root  operator  0x3a May 26 10:25 zfs
```

As you can see the mount shows that devfs is mounted twice. The second devfs mount somehow enforces the devices to be visible in the jail properly, as shown above.

This issue looked to me like a race condition so I tried to add some delays to exec.prestart but it didn't help. The devfs was still not being created properly. Anybody has any idea why this may be happening? Is having the devfs mounted twice as a workaround likely to cause any issues?

Greg


----------



## scottro (May 26, 2014)

There's a thread by me about this on the forums, which I'm too lazy to search out right now. I've filed a PR about this. feel free to add to it. 

I also mention it on my own page dealing with jails on nullfs at http://srobb.net/nullfsjail.html (a simpler, in my opinion, way to do it when compared to the handbook) mention the PR.  

Anyway, the PR is here. http://www.freebsd.org/cgi/query-pr.cgi?pr=186360


----------



## amiramix (May 26, 2014)

Thanks @scottro. Your page was one of the tutorials I was following and it's where I got the idea of using unionfs from  :beergrin . I didn't get to the bottom of it so I missed the part about FreeBSD 10 (next time pls please add the most recent stuff at the top). Good that there is already a PR for this. So now with my example at least you have a workaround to automatically mount devfs on boot


----------



## scottro (May 26, 2014)

Thanks, I'm going to add a note at the top about FreeBSD-10, (because I'm lazy), as well as point to this thread.


----------

