# Updated host to 13.0, now how to handle existing jails?



## jimbobmcgee (Aug 9, 2021)

I have recently bitten the bullet and taken my old 10.3 home lab server (HP Microserver N54L) up to 13.0.  I believe it went well -- certainly less painfully than I'd feared.

My lab runs off a USB boot/root, with some dirs (/usr/local, /usr/share, /usr/home, /var, /var) mounted from a ZFS pool.  I installed "fresh-ish" by building a new USB boot/root, using `gpart` to create a GPT on it, install the pmbr and gptboot bootcode, and `bsdinstall distextract` to unpack base.txz, lib32.txz and kernel.txz to it.  I did some jiggery with my ZFS mountpoints and sysutils/cpdup and have my system back to where it was pre-upgrade, using my old /var and /usr/home.  Finally, a `freebsd-update` to get my system up to 13.0-RELEASE-p3.

I took an awful lot of guidance from https://forums.freebsd.org/threads/...bout-installing-freebsd-on-a-usb-stick.11715/ and https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot -- many thanks to the writers of those; any mistakes are my own.

I haven't done a `zpool upgrade` yet so, technically, I can still put the old USB back in and boot my old 10.3 system, but I'm hoping not to have do that any more.

However, the meat of my FreeBSD usage is in single-purpose jails.  Those jails are built from old 10.1 and 10.3 templates, which I built from a scripted process, stashed in ZFS and made read-only.  Jail-specific fstab files mount the read-only template root, a matching read-write data filesystem (/usr/local, /var, /tmp, /etc, some others) and any other directories I need, all via nullfs.  Most applications I run from these jails were then installed via `pkg-static`.

Obviously any _new_ jail I create from here is going to use my newly-minted 13.0-RELEASE jail template, but (appreciating the _self-imposed_ complexity of my system) what is my best course of action regarding maintaining the _existing_ jails, particularly if I wish to resume `pkg` operations in them (either for updates or new features)?

*Should I alter the existing fstabs to mount my new 13.0-RELEASE template root in place of the old 10.x root?*  Will I have _a bad day(TM)_ with jails containing a read-only 13.0 root and 10.x writables?
*Should I  freebsd-update the existing 10.1 and 10.3 templates up to 13.0?*  Can I even _go_ up from 10.x to 13.0 via tools?  How might one _do_ an update to a "shared" read-only 10.x root?
*Should I do nothing?*  Is the host running 13.0 _enough_ (for some definition of "enough")
*Should I rebuild all the jails from scratch*  Hoping _not_ to have to do this, but am open to persuasion
*Some other action?*
*Some combination*
Have made a poll (since I'm asking a question), but discussion in the thread is very welcome.  Any particular direction for approaching the upgrade would be appreciated.


----------



## SirDice (Aug 9, 2021)

I'd try and migrate the jails and use a jail management tool like sysutils/bastille. That will make it much easier to keep the jails updated in the long run.


----------



## jimbobmcgee (Aug 9, 2021)

That's an interesting tool, which didn't exist when I first put my lab together, but ultimately has similar aims.  Long-term, that might well be an option, but I will need to read up on it first.

In the meantime, however, I'm still stuck as to how to update a series of existing shared/common-root 10.x jails to 13.0.  

Can `freebsd-update` even go from 10.x to 13.0?  If I do manage to get one of them updated, would a second one realise it had something to update?  If I just mount the 13.0 jail template, how would I drag up the /usr/local and /etc to 13.0?


----------



## SirDice (Aug 10, 2021)

jimbobmcgee said:


> which didn't exist when I first put my lab together


It's a fairly new tool, so that's entirely possible. On my host I used to use EZJail but that hasn't been updated in years and started to show its age. So now I'm migrating each jail one by one. You can have both EZJail and Bastille running, or in your case have the old 'handmade' jails running next to bastille. 



jimbobmcgee said:


> Can `freebsd-update` even go from 10.x to 13.0?


You may need to make sure that 10.x has the latest patch revision, if I recall correctly there was a bug in freebsd-update(8) that was fixed that prevented this. But yes, you normally should be able to go from 10.x to 13.0 in one step, there's no need to upgrade to any of the intermediate versions.



jimbobmcgee said:


> If I do manage to get one of them updated, would a second one realise it had something to update?


No, for all intents and purposes a jail is similar to a stand-alone server. If you upgrade one machine on your network the other machines don't have those updates either.



jimbobmcgee said:


> If I just mount the 13.0 jail template, how would I drag up the /usr/local and /etc to 13.0?


/usr/local is used by packages/ports, so you'll need to upgrade those too (you always have to reinstall everything after a major version upgrade anyway). As for /etc/, you can use mergemaster(8) or etcupdate(8).

Upgrading might be a rather slow and meticulous process. I would opt to create a new jail with bastille and just migrate the data from the old 'handmade' one to a new bastille jail. That way you can do this one by one and, if the bastille jail fails (because it wasn't set up properly for example) you can always start that old jail again. It's the same when you upgrade to a new system, you build it up next to the old system and make the switch when you're satisfied the new system is set up correctly.


----------



## jimbobmcgee (Aug 10, 2021)

SirDice said:


> No, for all intents and purposes a jail is similar to a stand-alone server. If you upgrade one machine on your network the other machines don't have those updates either.



...but if multiple jails share the same read-only base via nullfs mounts, then remounting one base read-write and updating _that_ affects all jails that use the same base.  But that means the first jail is fully-updated, and the others are only _partially_ updated.  How would a tool like `freebsd-update` handle that in the _second_ jail I updated.

From the looks of https://github.com/BastilleBSD/bastille/blob/master/usr/local/share/bastille/upgrade.sh, even Bastille won't handle this (`bastille upgrade` only appears to work for Bastille-termed "thick" jails which, as far as I can tell, are literal copies (or `zfs send | zfs receive` of a template release.



SirDice said:


> /usr/local is used by packages/ports, so you'll need to upgrade those too (you always have to reinstall everything after a major version upgrade anyway). As for /etc/, you can use mergemaster(8) or etcupdate(8).



Appreciating that you haven't said to use `freebsd-update` anywhere, I vaguely recall being told never to mix `freebsd-update` and `mergemaster`.  Am I misremembering that, or does that still stand?



SirDice said:


> You may need to make sure that 10.x has the latest patch revision, if I recall correctly there was a bug in freebsd-update(8) that was fixed that prevented this. But yes, you normally should be able to go from 10.x to 13.0 in one step, there's no need to upgrade to any of the intermediate versions.



Don't suppose you remember the details of that bug, or where I might read up about it?  Trying to determine if it's the same one I'm seeing...

As a test, I've taken one of my less important jails and `zfs clone`-ed the shared template root into its own filesystem, so it is now technically entirely independent.  I've run `freebsd-update -r 13.0-RELEASE --currently-running 10.3-RELEASE-p16 upgrade install` and can see a few thousand patches and files fetched, but the install doesn't seem to complete -- at least `$status` is 1 and, after restarting the jail, `freebsd-version` still reports 10.3-RELEASE-p16, so I assume it hasn't completed.

During the Install phase, I see a bunch of _Operation not permitted_ messages:


```
Installing updates...chflags: ///bin/rcp: Operation not permitted
chflags: ///lib/libc.so.7: Operation not permitted
chflags: ///lib/libcrypt.so.5: Operation not permitted
chflags: ///lib/libthr.so.3: Operation not permitted
chflags: ///libexec/ld-elf.so.1: Operation not permitted
...
```

...which I guess are due to these being on ZFS, rather than UFS filesystems.  If that's not the issue, is there any logging/verbosity I can apply to see exactly why the `freebsd-update` is not completing?  I don't see a `-v` option.


----------



## SirDice (Aug 11, 2021)

jimbobmcgee said:


> ...but if multiple jails share the same read-only base via nullfs mounts, then remounting one base read-write and updating _that_ affects all jails that use the same base. But that means the first jail is fully-updated, and the others are only _partially_ updated. How would a tool like `freebsd-update` handle that in the _second_ jail I updated.


In that case you only update that base jail. As all the other jails are using the base jail they're all upgraded at the same time. 



jimbobmcgee said:


> From the looks of https://github.com/BastilleBSD/bastille/blob/master/usr/local/share/bastille/upgrade.sh, even Bastille won't handle this (`bastille upgrade` only appears to work for Bastille-termed "thick" jails which, as far as I can tell, are literal copies (or `zfs send | zfs receive` of a template release.


No, the thin jails use a similar construct. You update the base jail and that upgrades all jails built from that base jail.

```
root@armitage:~ # mount | grep bastille | grep games
zroot/bastille/jails/games on /bastille/jails/games (zfs, local, noatime, nfsv4acls)
zroot/bastille/jails/games/root on /bastille/jails/games/root (zfs, local, noatime, nfsv4acls)
/bastille/releases/13.0-RELEASE on /bastille/jails/games/root/.bastille (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /bastille/jails/games/root/dev (devfs)
root@armitage:~ # ll /bastille/jails/games/root/
total 174
drwxr-xr-x  18 root  wheel    22 Apr 25 19:25 .bastille/
-rw-r--r--   1 root  wheel  1023 Apr  9 08:17 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  9 08:17 .profile
drwxr-xr-x   2 root  wheel     2 May  8 18:00 .template/
-r--r--r--   1 root  wheel  6109 Apr  9 08:26 COPYRIGHT
lrwxr-xr-x   1 root  wheel    14 May  8 18:00 bin@ -> /.bastille/bin
lrwxr-xr-x   1 root  wheel    15 May  8 18:00 boot@ -> /.bastille/boot
dr-xr-xr-x  10 root  wheel   512 Jul 10 18:30 dev/
drwxr-xr-x  27 root  wheel   108 May  9 13:15 etc/
lrwxr-xr-x   1 root  wheel     8 May  8 18:00 home@ -> usr/home
lrwxr-xr-x   1 root  wheel    14 May  8 18:00 lib@ -> /.bastille/lib
lrwxr-xr-x   1 root  wheel    18 May  8 18:00 libexec@ -> /.bastille/libexec
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 media/
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 mnt/
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 net/
dr-xr-xr-x   2 root  wheel     2 Apr  9 08:09 proc/
lrwxr-xr-x   1 root  wheel    17 May  8 18:00 rescue@ -> /.bastille/rescue
drwxr-x---   2 root  wheel     9 Jul 21 21:36 root/
lrwxr-xr-x   1 root  wheel    15 May  8 18:00 sbin@ -> /.bastille/sbin
drwxrwxrwt   7 root  wheel     7 Aug 11 03:01 tmp/
drwxr-xr-x   6 root  wheel    15 May  8 18:05 usr/
drwxr-xr-x  24 root  wheel    24 Jul 10 18:30 var/
```



jimbobmcgee said:


> Don't suppose you remember the details of that bug, or where I might read up about it? Trying to determine if it's the same one I'm seeing...


It was this one: https://www.freebsd.org/security/advisories/FreeBSD-EN-16:09.freebsd-update.asc



jimbobmcgee said:


> I've run `freebsd-update -r 13.0-RELEASE --currently-running 10.3-RELEASE-p16 upgrade install` and can see a few thousand patches and files fetched, but the install doesn't seem to complete


You'll need to run `freebsd-update install` three times. First run will only update the kernel, second run will update the userland and the third run will remove old libraries and other files.


----------



## jimbobmcgee (Aug 15, 2021)

SirDice said:


> You'll need to run `freebsd-update install` three times. First run will only update the kernel, second run will update the userland and the third run will remove old libraries and other files.



I couldn't even get `freebsd-update install` to complete _once_.  I saw a bunch of _Operation not permitted_ messages.  I eventually worked out that this was because I was trying to run it _inside_ my jail, which prevents the `chflags` operations that `freebsd-update` does internally.  Not sure if that is documented, but I didn't discover it via documentation, so it might be worth updating chflags(1) and freebsd-update(8) to call this out explicitly.

Once I mounted all my jail filesystems in the right order/to the right mountpoints, and issued `freebsd-update -b /pathto/jail ...` from the _host_, my updates ran OK.

____



SirDice said:


> No, the thin jails use a similar construct. You update the base jail and that upgrades all jails built from that base jail.



Yes, on re-read of the Bastille source, I can see that now. 

But I still can't see how that then updates the contents of /etc, which is part of the thin jail filesystem and _not_ the release filesystem.

A _thick_ jail is obvious to me -- since it's a straight copy of the release, `freebsd-update -b /pathto/jail --currently-running 10.1-RELEASE` can update everything in the copy -- and this appears to be what `bastile upgrade mythickjail 13.0-RELEASE` does.  A _thin_ jail looks to have the same problem that I currently have with my hand-cranked thin jail; where, after `bastile upgrade 10.1-RELEASE 13.0-RELEASE`, the readonly-mounted release (symlinked from /.bastille) is updated to 13.0-RELEASE, but the read-write mounted /etc still has the 10.1-RELEASE versions of the config.

I've briefly looked at mergemaster(8) and etcupdate(8), which look like they apply here, but I can't see how you specify (or how the command otherwise determines) the _old_ version for the three-way merge (i.e. there is no obvious equivalent `--currently-running` argument, like there is in `freebsd-update`).


----------



## Brian546 (Aug 15, 2021)

I don't think freebsd-update has options to only update the world or only update /etc. If it did though I think it would make it easier to update nullfs mounted base jails properly. I seem to remember receiving the same errors you were getting, and it only updated the shared world of the jails, not any of the rw files from each jail.

At least by compiling the source you can make installworld to your base folder, and then mergemaster or etcupdate to your rw folders. But I know you're trying to use freebsd-update. Not sure there is a way.


----------

