# Jail root nullfs won't unmount Device Busy



## geekspark (Apr 4, 2014)

I recently upgraded my server from FreeBSD 9.0 to FreeBSD 10.0. After the upgrade I converted all of my old jails from using /etc/rc.conf to using /etc/jail.conf. However when I attempt to stop some jails they refuse to unmount their root filesystem which is a read-only nullfs() mount that points to a shared FreeBSD 10.0 world. All jails start and run correctly the only issue is at shutdown, and only occurs for some jails.

Output from running `jail -r -v www1`

```
www1: run command in jail as root: /bin/sh /etc/rc.shutdown
www1: removed
www1: run command as root: /sbin/umount /mnt/disk1/j/10.0.3.1-www1/dev
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/tmp
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/var
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/root
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/home
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/ports/portbuild
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/ports/distfiles
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/X11R6
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/local
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/etc
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1
umount: unmount of /mnt/disk1/j/10.0.3.1-www1 failed: Device busy
jail: www1: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1: failed
www1: run command as root: /sbin/ifconfig ale0 inet 10.0.3.1 netmask 255.255.255.255 -alias
```

Everything appears to be unmounting in the proper order, but the root mount just refuses. `umount -f /mnt/disk1/j/10.0.3.1-www1` will remove the mount, and waiting about 2 minutes will allow me to do a nonforced umount. fstat() also shows nothing being accessed in the root directory.

One jail that refuses to umount its root was upgraded from a FreeBSD 9.0 built world to a FreeBSD 10.0 built world and has gone through the complete upgrade process of using mergemaster() and upgrading all ports. Another jail was newly created using only a FreeBSD 10.0 built world without requiring any upgrades to ports or files since it was up-to-date already. Yet both jails fail to unmount their root mounts.

I'm at an extreme loss as to why this is happening and have used google and searched the forums but can only find people with the similar issues but all tend to be related to files being open and displayed when using fstat(), however in my case fstat() shows nothing even if told to show memory mapped files.

Edit: also here's the output of `uname -a` just to be clear to show the version, kernel, and architecture if that's relevant

```
FreeBSD host1.localdomain.local 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014     root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```


----------



## mistige (Jan 3, 2019)

2019 now,
The devfs is mounted, which prevents the 
umount of the nullfs
You can try to umount the devfs first:

```
# jail -c testjail    # testjail has run earlier  of the host
mount_nullfs: /usr/jail/testjail: Resource deadlock avoided
jail: testjail: /sbin/mount -t nullfs -o rw /usr/jail/basejail /usr/jail/testjail: failed
# umount /usr/jail/testjail/dev          ### umount devfs
# jail -c testjail
mount_nullfs: /usr/jail/testjail: Resource deadlock avoided
jail: testjail: /sbin/mount -t nullfs -o rw /usr/jail/basejail /usr/jail/testjail: failed
# umount -t nullfs /usr/jail/testjail      ### umount nullfs
# mount | grep nullfs                     ###  check all clear
# jail -c testjail
testjail: created                  ### works again
# jls
   JID  IP Address      Hostname                      Path
    10  192.168.0.10    testjail                      /usr/jail/testjail
```


----------



## SirDice (Jan 3, 2019)

Order in fstab.<jailname> is just as important as for the system's fstab. You cannot unmount a filesystem that still has other mounts underneath it.


----------



## Roy Chan (Feb 2, 2019)

Same issue here. Sill have no idea...


----------



## ShelLuser (Feb 2, 2019)

fuser(1) is your friend here. Just check what pid(s) are accessing the mountpoint and stop those processes.


----------



## Roy Chan (Feb 2, 2019)

ShelLuser said:


> fuser(1) is your friend here. Just check what pid(s) are accessing the mountpoint and stop those processes.



Yes I could use `fstat` or `fuser` to check the usage of a specific mount point. However, after the `device busy` error, I can umount the base file system from the host without this error. And `fstat` shows no opened files. This error occurs approximately 1 time per 10 times start/stop of this jail. Weird.


----------



## Roy Chan (Feb 2, 2019)

Update: I added `exec.poststop = "sleep 1"` to this jail and it seems that the problem disappears. Maybe the mounted base file system is still busy right after the remove of the jail?


----------



## ShelLuser (Feb 3, 2019)

It is possible I suppose, though I don't have those experiences.

Still: _how_ do you mount those in the first place?  I assume you're using mount.fstab in /etc/jail.conf?


----------



## Roy Chan (Feb 4, 2019)

ShelLuser said:


> It is possible I suppose, though I don't have those experiences.
> 
> Still: _how_ do you mount those in the first place?  I assume you're using mount.fstab in /etc/jail.conf?



For me, yes. (Why our replies were moved here... I thought it should be here: https://forums.freebsd.org/threads/...metimes-not-un-mounted-after-jail-stop.69415/)


----------

