# Jail lost all packages



## pming (May 19, 2017)

Hello everyone

I recently set up Nextcloud in a jail using apache24, mariadb101 and php70 on FreeBSD 10.3.
After rebooting my server the jail will not come up correctly. It won't mount datasets and all the packages I installed and their configuration seems gone. `pkg info` only returns pkg.

I used the following scripts to automount datasets when the jail starts:

http://vedad.kajtaz.net/perso/ezjail/rc.boot.zfs
http://vedad.kajtaz.net/perso/ezjail/rc.mount.zfs
http://vedad.kajtaz.net/perso/ezjail/rc.shutdown.zfs

Here's my /etc/jail.conf:

```
# /etc/jail.conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
host.hostname = "$name";
mount.devfs;
path = "/zroot/jails/$name";

nextcloud {
    persist = true;
    exec.poststart = "/bin/sh /usr/local/bin/jail_scripts/nextcloud/jail_datasets.sh";
    exec.start = "/bin/sh /usr/local/etc/jails/rc.boot.zfs zroot/jails/nextcloud/zroot zroot/jails/nextcloud/db tank/nextcloud/data";
    exec.stop = "/bin/sh /usr/local/etc/jails/rc.shutdown.zfs";
    exec.poststop = "/bin/sh /usr/local/bin/jail_scripts/nextcloud/unjail_datasets.sh";
    allow.mount = true;
    allow.mount.zfs = true;
    enforce_statfs = 1;
    interface = "ixl0.50";
    ip4.addr = "192.168.50.41";
}
```

The poststart and poststop scripts will `/sbin/zfs jail nextcloud <dataset>` or unjail them respectively.

Stopping the jail takes forever. If I restart it and execute `mount` on the host, I see the line


```
devfs on /zroot/jails/nextcloud/dev (devfs, local, multilabel)
```

as many times as I restarted the jail. Same thing inside the jail. The data itself (Nextcloud data and database) still lives in the corresponding datasets.

Any suggestions are welcome.


Best Regards
Philipp


----------



## SirDice (May 19, 2017)

Your start script never executes /etc/rc so the jail is never actually booted. Also note that poststart is executed _after_ start, you probably want to start it _before_.

The order is:

```
prestart
start
poststart
--
prestop
stop
poststop
```


----------



## pming (May 19, 2017)

I am pretty sure /etc/rc should get executed as soon as the necessary datasets are available to the jail, because rc.boot.zfs calls rc.mount.zfs, which in turn starts /etc/rc.

Here's the scripts so we can look them up easier:

rc.boot.zfs

```
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

daemon -f /usr/local/etc/jails/rc.mount.zfs "$@"
exit 0
```

rc.mount.zfs

```
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

# What we do here is:
# 1. Wait for jailed zfs to pop-up - which unfortunately doesn't happen at real boot time due to /etc/rc.d/jail weirdness
# 2. If mount is allowed, also mount the datasets
#    For the mount to be allowed, all of the following jail parameters need to be set:
#    * enforce_statfs=1 (or enforce_statfs=0)
#    * allow.mount.zfs=1
#    However, in order to manage the dataset without ability to mount/unmount (handy on backup jails where we MUST NOT mount the received datasets), the following is enough/required:
#    * enforce_statfs=1 (or enforce_statfs=0)


test_zfs_datasets()
{
   for i in "$@"; do
        if [ "`/sbin/zfs list -H -o name $i 2>/dev/null`" != "$i" ]; then
            return 1
        fi
    done

    return 0
}

while [ 1 ]; do
    sleep 1.5
    test_zfs_datasets "$@"
    if [ "$?" -eq 0 ]; then
        if [ `/sbin/sysctl -n security.jail.mount_zfs_allowed` -eq 1 -a `/sbin/sysctl -n security.jail.enforce_statfs` -lt 2 ]; then
            /sbin/zfs unmount -a 2>/dev/null
            /sbin/zfs mount -a 2>/dev/null
        fi

        /bin/sh /etc/rc
        exit 0
    fi
done
```

rc.shutdown.zfs

```
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

/bin/sh /etc/rc.shutdown
_SHUTDOWN_RET=$?

/sbin/zfs unmount -a 2>/dev/null

exit $_SHUTDOWN_RET
```

I stumbled upon them on this post.

As the man page for jail says,

```
exec.poststart
        Command(s)    to run in the system environment after a jail is cre-
         ated, and after any exec.start commands have completed.
```
I think it's possible the poststart script does not get executed, because the exec.start script does not complete successfully (because the datasets need to be available for that).

I tested with prestart too, but no datasets will get mounted either. Using prestart, the datasets don't even get attached and `zfs list` inside the jail gives me:

```
no datasets available
```

I don't know why this should stop working after a reboot. Before that I could restart the jail without any problems. Also configuration files and rc scripts in /usr/local/etc for the packages are gone. Where else would a jail even save such files? Using the configuration in my first post, `mount` in the jail shows the root is mounted:


```
zroot/jails/nextcloud on / (zfs, local, noatime, nfsv4acls)
```


----------

