# How to create a ZFS dataset within a jail?



## ikevin8me (Aug 26, 2017)

I'm trying to run Docker (must be within a FreeBSD jail)

How do I create this:

```
--
You will need to create a ZFS dataset on /usr/docker
# zfs create -o mountpoint=/usr/docker <zroot>/docker
--
```

when `zfs list` shows "no datasets available" ?

The jail name is "testing" and it seems that is trying to detect "zroot/jails/testing".


```
--
# docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem zroot/jails/testing: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist
--
```

Please help. Thanks!


----------



## ComradeSlice (Aug 26, 2017)

Do you have mount.devfs; in /etc/jail.conf? I used to have a ZFS jail and I believe that was necessary for ZFS to function in the jail. What are the values of these sysctl variables?

```
security.jail.enforce_statfs
security.jail.mount_allowed
security.jail.mount_devfs_allowed
security.jail.mount_zfs_allowed
```


----------



## ikevin8me (Aug 27, 2017)

I'm still using the old style rc.conf, I can't find the equivalent for:

```
security.jail.enforce_statfs
security.jail.mount_zfs_allowed
```
It would be nice if you could tell me how to do it in rc.conf.

For these, the equivalent are:
security.jail.mount_allowed =   allow.mount    set from _jail_*_*<_jname_>*_*_mount_*_*_enable_
security.jail.mount_devfs_allowed =   mount.devfs    set from _jail_*_*<_jname_>*_*_devfs_*_*_enable
_
*Thanks!*


----------



## ikevin8me (Aug 27, 2017)

My current values are:
security.jail.enforce_statfs: 2
security.jail.mount_allowed: 0
security.jail.mount_devfs_allowed: 0
security.jail.mount_zfs_allowed: 0


----------



## ComradeSlice (Aug 28, 2017)

Yeah you're going to need those sysctl values turned on. Can't do it in rc.conf. Do this in sysctl.conf:


```
security.jail.mount_allowed=1
security.jail.mount_devfs_allowed=1
security.jail.mount_zfs_allowed=1
```

If you want to be able to ping from within your jail, turn this on as well:


```
security.jail.allow_raw_sockets=1
```

Also add this to your jail.conf file. At the top, not in the jail definition:


```
mount.devfs;
```

Let me know if this helps.


----------



## ikevin8me (Sep 1, 2017)

I tried to change these values in the *sysctl.conf *and now [the global zone] shows:

```
# sysctl security.jail.mount_allowed
security.jail.mount_allowed: 1
# sysctl security.jail.mount_devfs_allowed
security.jail.mount_devfs_allowed: 1
# sysctl security.jail.mount_zfs_allowed
security.jail.mount_zfs_allowed: 1
```
From inside the jail, it is (I don't think this is relavant, but just in case):

```
# sysctl security.jail.mount_allowed
security.jail.mount_allowed: 0
# sysctl security.jail.mount_devfs_allowed
security.jail.mount_devfs_allowed: 0
# sysctl security.jail.mount_zfs_allowed
security.jail.mount_zfs_allowed: 0
```

The jail always had the "jail_testing_devfs_enable="YES" in the rc.conf.

The "security.jail.enforce_statfs: 2" remained as 2. I did not change this. Do I need to change this to 0?

Alright, here is the problem: I'm trying to run Docker within a jail and I got:
*"filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist" error"*

It seems that I have to make 'zroot/jails/testing': dataset to exist from within the jail. How?

I have tried "zfs create -o mountpoint=/usr/docker zroot/docker" and it says *"no such pool"*.

I also tried (from inside the jail):
# dd if=/dev/zero of=/usr/local/dockerfs bs=1024K count=4000
# zpool create -f zroot /usr/local/dockerfs
*cannot create 'zroot': permission denied*

Well, I'm trying to run Docker inside a jail. I believe that once resolve the ZFS dataset issue, docker will run.


----------



## ComradeSlice (Sep 2, 2017)

Did you add `mount.devfs;` to your `/etc/jail.conf`? Show me the output of the `mount` command from within the jail.


----------



## ikevin8me (Sep 2, 2017)

Yes, I'm still using the old-style rc.conf and 'jail_testing_devfs_enable="YES"' had always been there.

The output from within the jail:
# mount
zroot/jails/testing on / (zfs, local, noatime, nfsv4acls)

More info (all from inside the jail):
# docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem zroot/jails/testing: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all zroot/jails/testing" => cannot open 'zroot/jails/testing': dataset does not exist
# zfs list
no datasets available


----------



## ComradeSlice (Sep 2, 2017)

Try mounting devfs from outside the jail, then restart the jail.

`mount -t devfs devfs /jails/testing/dev`

Let me know if this works.


----------



## ikevin8me (Sep 2, 2017)

Yes, that worked. I mounted and I can access "/dev" from within the "testing" jail.

From inside the jail, it shows:

root@testing:/dev # ls -la
total 10
dr-xr-xr-x   9 root  wheel      512 Sep  2 05:18 .
drwxr-xr-x  18 root  wheel       22 Aug 24 03:13 ..
dr-xr-xr-x   2 root  wheel      512 Sep  2 05:18 fd
crw-rw-rw-   1 root  wheel     0x1b Sep  2 05:18 null
dr-xr-xr-x   2 root  wheel      512 Sep  2 05:18 pts
crw-r--r--   1 root  wheel      0x7 Sep  2 02:15 random
lrwxr-xr-x   1 root  wheel        4 Sep  2 05:18 stderr -> fd/2
lrwxr-xr-x   1 root  wheel        4 Sep  2 05:18 stdin -> fd/0
lrwxr-xr-x   1 root  wheel        4 Sep  2 05:18 stdout -> fd/1
lrwxr-xr-x   1 root  wheel        6 Sep  2 05:18 urandom -> random
crw-rw-rw-   1 root  wheel     0x1c Sep  1 21:15 zero
crw-rw-rw-   1 root  operator  0x4e Sep  2 02:15 zfs
root@testing:/dev #

How can I resolve this problem (from within the jail):
# zpool create -f zroot /usr/local/dockerfs
*cannot create 'zroot': permission denied*


----------



## ComradeSlice (Sep 3, 2017)

Sorry, I know you asked but I forgot to mention `security.jail.enforce_statfs` should be 0. That will allow the jail to see the mounted filesystems.

EDIT: I do not think you actually have to do what you are trying to do. Docker relies on ZFS but you only have to create that vdev backed pool if your system does not already use ZFS. Let's stick to trying to get the jail to see ZFS and not trying to create that pool.


----------



## ComradeSlice (Sep 3, 2017)

I'm setting up my own jail to troubleshoot. Should have an answer shortly.


----------



## ComradeSlice (Sep 3, 2017)

This, on top of the other options we discussed, should do the trick 
I don't remember having to do this with my last jail but I just set one up and in addition to the sysctl values and mounting devfs this must be done.

This shows the given dataset (zroot/jails/testing) to the given jail.

`zfs jail [jail name] zroot/jails/testing`


----------



## ikevin8me (Sep 4, 2017)

Thanks! I went past that stage. The dataset is now accessible within the jail.

However, I've got into another problem:

# service docker onestart
Starting docker...
netstat: kvm not available: /dev/mem: No such file or directory

I tried to find out but can't figure out how to mount this "/dev/mem" into the jail. Please help. Appreciated.


----------



## ComradeSlice (Sep 4, 2017)

That's one of the memory files. In particular, the one that maps physical memory to file offsets. It's required for kvm to function. kvm is a library for accessing kernel data. Netstat uses kvm. This is normal for jails, as I understand it. Same thing happens to me:

`[root@build11 /]# netstat
netstat: kvm not available: /dev/mem: No such file or directory`

Netstat still gives me output though. Does this actually prevent docker from running?

Jails interface with the host's kernel through purpose built calls. Jails don't have their own kernel. You don't want a jail having physical memory access. The design of a jail should not allow the root of a jail to take over the physical machine.


----------



## ikevin8me (Sep 4, 2017)

Yes, it prevents Docker from running.

I execute "service docker onestart", it showed the error message and Docker stops dead. I look into "top" and did not see any Docker process running. (I am following this.)


----------



## ComradeSlice (Sep 4, 2017)

Run both of these commands in your host system and in your jail. Post all four lines of output and tell me which pair came from the jail and which from the host.

`uname -U
uname -K`


----------



## ikevin8me (Sep 5, 2017)

From host:
# uname -U
1101001
# uname -K
1101001

From jail:
root@testing:/ # uname -U
1101001
root@testing:/ # uname -K
1101001


----------



## ComradeSlice (Sep 5, 2017)

Please paste in Docker's rc.d script. That'll help me figure out how to start it without daemonizing and with a bit more verbosity. I don't think netstat is the issue. That is the normal output from within a jail. It's _impossible _to use KVM from within a jail. Your kernel, userland, and jail are all in sync. Something else is causing the problem.


----------



## ikevin8me (Sep 6, 2017)

I have uploaded the file (attached). Thanks!


----------



## ComradeSlice (Sep 8, 2017)

You can stop docker from using netstat if you set the `docker_nat_iface` variable in your rc.conf. Run `netstat -f inet -nrW | grep '^default' | awk '{ print $6 }'` in the host environment to get the interface and set `docker_nat_iface` to that interface. For me it would be `docker_nat_iface="em0"`


----------



## ikevin8me (Sep 9, 2017)

By putting 'docker_nat_iface="em0"' into the rc.conf of the jail, I resolved the problem "netstat: kvm not available: /dev/mem: No such file or directory" (it does not show up anymore).

Docker is still failing to start because the ZFS dataset at location /usr/docker cannot be found by /proc/mount.

Well, I think I give up and would not like to try further to run Docker in a jail. It had been a great frustration. Thanks for your help very much and I'm sorry to disappoint that it does not work.


----------



## ikevin8me (Sep 9, 2017)

I have one more question:

What exactly does this 'mount -t devfs devfs /jails/testing/dev' do? It seems that it has been mounted on more than the "testing" jail (ie. affected the other jails as well). 

How do I undo this mount?


----------



## ComradeSlice (Sep 9, 2017)

It's not persistent, unless you put that in fstab. Just `umount /jails/testing/dev`.


----------



## NetSoerfer (Jun 23, 2018)

/edit: turns out I forgot to set `enforce_statfs=0`. I'm beyond the below error now, but getting another error:


```
root@docker:/ # docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem freenas-boot/ROOT/11.1-U5: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all freenas-boot/ROOT/11.1-U5" => cannot open 'freenas-boot/ROOT/11.1-U5': dataset does not exist.
```

I'll report back after investigating this further.



Spoiler: Original Post






ComradeSlice said:


> This, on top of the other options we discussed, should do the trick
> I don't remember having to do this with my last jail but I just set one up and in addition to the sysctl values and mounting devfs this must be done.
> 
> This shows the given dataset (zroot/jails/testing) to the given jail.
> ...


Sorry to revive such an old thread, but this seems pretty similar to the problems I'm fighting with at the moment.

I'm trying to get docker to run in an iocage jail. I believe I've translated the `security.jail.mount_allowed=1`, `security.jail.mount_devfs_allowed=1`, `security.jail.mount_zfs_allowed=1`, and `mount.devfs;` to their iocage setting equivalents, but now I'm stuck because I don't know how to show the dataset to the jail, or how to show the right one.

After running `zfs jail 28 tank/iocage/jails/docker`, `docker -d` still results in the following output:


```
root@docker:/ # docker -d
WARN[0000] Kernel version detection is available only on linux
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
FATA[0000] Error starting daemon: error initializing graphdriver: Cannot find root filesystem tank/iocage/jails/docker/root: exit status 1: "/sbin/zfs zfs get -rHp -t filesystem all tank/iocage/jails/docker/root" => cannot open 'tank/iocage/jails/docker/root': dataset does not exist
```


The error actually says that the ./docker/*root *dataset does not exist. That's true, because at the moment that is a directory within the ./docker dataset. Does docker actually require this to be a *separate* dataset within ./docker? Do I have to create the ./docker/root dataset manually _before _creating the iocage jail? (That just seems a bit odd to me since the ./docker dataset is usually created automatically along with the jail, hence my question)


----------



## NetSoerfer (Jul 1, 2018)

I discussed the issue in the FreeNAS forums, feeling that it might be closer related to FreeNAS than FreeBSD. That discussion ultimately led me to give up on docker in jails, in favour of docker in a VM. Below are the points that settled it for me:


			
				Ericloewe said:
			
		

> Every Docker image or whatever out there is Linux software. FreeBSD could have 100% compatibility with the management layer, but that still wouldn't help anyone since the software you want to run requires Linux.
> If FreeBSD ends up supporting all the Linux stuff, like Smart OS does in Illumos-land, it would be possible to run all that stuff on FreeBSD, but you'd still be tied to some of the Linux-isms (besides the system call layer, which isn't _too_ hard to do)
> Without such support, you need a Linux VM or something else that is very good at pretending to be Linux (Hello Windows and Smart OS!)


----------

