# bectl & upgrade with full rollback ability



## Selin (Mar 26, 2021)

Hi

I'd like to figure out how to do an upgrade with the ability to rollback fully if anything goes wrong.
Taking into account the info from the following topics:

https://forums.freebsd.org/threads/...ke-snapshots-and-do-rollbacks-of-zroot.61297/
https://forums.freebsd.org/threads/upgrade-freebsd-with-zfs-boot-environments.79025/
https://forums.freebsd.org/threads/adventures-in-upgrading-boot-environments.77032/
*Initial system state*


Spoiler





```
> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed
```




I do the following:

`zfs snap -r zroot@original`
`bectl create -r new`
`bectl mount new`
`chroot {path to mounted "new" snapshot}`
`mount -t devfs devfs /dev`
`rm -rf /var/db/freebsd-update`
`mkdir /var/db/freebsd-update`
`freebsd-update upgrade -r {NEW_VERSION}`
`freebsd-update install`
`bectl activate new`
`reboot`
What I have at this point


Spoiler





```
> uname -a
FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

> freebsd-version -kru
13.0-RC3
13.0-RC3
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed
```




`freebsd-update install`
rebuild/update _all ports/packages installed_
`freebsd-update install`
_ gpart bootcode -p /boot/boot1.efifat -i 1 nvd0_
`reboot`
*Final System state*


Spoiler





```
> uname -a
FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

> freebsd-version -kru
13.0-RC3
13.0-RC3
13.0-RC3

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.21           GNU gettext runtime libraries and programs
gettext-tools-0.21             GNU gettext development and translation tools
glib-2.66.7_1,1                Some useful routines of C programming (current stable version)
gmake-4.3_2                    GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libiconv-1.16                  Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.21              Text styling library
mc-nox11-4.8.26                Midnight Commander, a free Norton Commander Clone
meson-0.57.1                   High performance build system
ninja-1.10.2,2                 Small build system closest in spirit to Make
pcre-8.44                      Perl Compatible Regular Expressions library
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
pkgconf-1.7.4,1                Utility to help to configure compiler and linker flags
py37-setuptools-44.0.0         Python packages installer
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed
```




Here I have my system upgraded and what to rollback to the initial state (including kernel and versions of all installed ports).
So I'm going to do the following:

`zfs snap -r zroot@new`
`bectl activate default`
`reboot` # here I'm booting from LiveCD
_import and mount my zpool_
`cd {mount dir}`
`mkdir backup`
`mv bin boot etc lib libexec rescue sbin usr var backup/`
`cp -pR {mount dir}/.zfs/snapshot/original/. {mount dir}`
_ gpart bootcode -p {mount dir}/boot/boot1.efifat -i 1 nvd0_
*Rollback result*


Spoiler





```
> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed
```




Now it seems the system is the same as I had before the upgrade.
Is that correct?
Or I should adjust them somehow?

If some user files were updated in a new system - they can be copied from the `/.zfs/snapshot/new/usr/home/{user}`

And what about this part of the upgrade/rollback?
`gpart bootcode -p /boot/boot1.efifat -i 1 nvd0`
Is that correct to update/rollback UEFI loader that way?

Thanks


----------



## zirias@ (Mar 26, 2021)

I only had a quick look now, but doing a global snapshot of `zroot` looks a bit over the top to me. Boot environments already handle the base system. If you also upgrade ports/packages and /usr/local is on a _different_ dataset (which isn't the case in the structure created by the installer), just add a snapshot there as well.

As for things like /var and /usr/home or /home, you should probably think twice whether you want to roll them back at all. Normally, a boot environment will be enough. As long as you don't upgrade your pool (you shouldn't unless you're SURE you don't want to go back), bootcode shouldn't be an issue either, but if you already updated it, just rewrite it from your other BE to go back.


----------



## SirDice (Mar 26, 2021)

Note that with 13.0 that boot1.efifat file doesn't exist anymore. You need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi (FAT is not case-sensitive, so actual case doesn't matter) of that filesystem.

There shouldn't be a need to rollback the EFI loader, it's backwards compatible. You can boot 12.x with the 13.0 loader.efi but not the other way around.


----------



## Selin (Mar 26, 2021)

I need to rollback the whole system state: kernel + set of ports installed + all their configs, etc.
Sometimes it might be needed to rollback some application config and data. E.g. DB files might be converted to a new format after DB engine upgrade.
So I decided to do a full backup.

Here is the ZFS structure I have with default installation:

```
> zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               6.24G  6.84G    88K  /zroot
zroot/ROOT          4.00G  6.84G    88K  none
zroot/ROOT/default  2.47G  6.84G  1.89G  /
zroot/tmp            336K  6.84G   120K  /tmp
zroot/usr           2.22G  6.84G    88K  /usr
zroot/usr/home       268K  6.84G   128K  /usr/home
zroot/usr/ports     1.53G  6.84G   880M  /usr/ports
zroot/usr/src        704M  6.84G   704M  /usr/src
zroot/var            932K  6.84G    88K  /var
zroot/var/audit       88K  6.84G    88K  /var/audit
zroot/var/crash       88K  6.84G    88K  /var/crash
zroot/var/log        420K  6.84G   156K  /var/log
zroot/var/mail        88K  6.84G    88K  /var/mail
zroot/var/tmp        160K  6.84G    88K  /var/tmp
```

`bectl -r ...` creates snapshot only for the `zroot/ROOT/default` volume.

What about `zroot/usr` - isn't it affected during upgrading the system and ports?


----------



## zirias@ (Mar 26, 2021)

I wonder how your /usr ended up containing anything, it should normally be set to `canmount=no`, so everything in /usr (except for the specific datasets you create below) will be stored on your root dataset.

*edit:* forget the nonsense, looked at the wrong column. Your `zroot/usr` seems to be empty, as expected.


----------



## SirDice (Mar 26, 2021)

Selin said:


> What about `zroot/usr` - isn't it affected during upgrading the system and ports?


It's not mounted, it's only there to provide easy access to for /usr/home, /usr/ports and /usr/src. The actual data in /usr/ is stored within zroot/ROOT/default.

```
root@molly:~ # zfs get canmount zroot/usr
NAME       PROPERTY  VALUE     SOURCE
zroot/usr  canmount  off       local
```

/usr/ports doesn't need to be rolled back (or forward), all versions use the same ports tree. /usr/src/ doesn't really matter either, since it's version controlled you can more easily deal with that using svnlite(1) or git(1) (just switch branches).


----------



## Selin (Mar 29, 2021)

Thanks everybody for advices.

Updated my notes to the following:

*Upgrade*

`bectl create -r {new}`
`bectl mount {new}`
`chroot {path to mounted "new" snapshot}`
`mount -t devfs devfs /dev`
`rm -rf /var/db/freebsd-update`
`mkdir /var/db/freebsd-update`
`freebsd-update upgrade -r {NEW_VERSION}`
`freebsd-update install`
`bectl activate {new}`
`reboot`
`freebsd-update install`
rebuild/update _all ports/packages installed_
`freebsd-update install`
_ gpart bootcode -p /boot/boot1.efifat -i 1 nvd0_
`reboot`
*check if everything works good because after the next step there will be no ability to rollback*
`zpool upgrade -a`
*Rollback*

`bectl activate {old}`
_ chroot {path to mounted "old" snapshot}_
_ mount -t devfs devfs /dev_
_ gpart bootcode -p /boot/boot1.efifat -i 1 nvd0_
`reboot`
*NOTE*
In 13.0 the boot1.efifat file doesn't exist anymore.
So instead of _ gpart bootcode -p /boot/boot1.efifat -i 1 nvd0 _need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi


----------



## mtu (Mar 29, 2021)

Selin said:


> *Upgrade*
> 
> `bectl create -r {new}`
> `bectl mount {new}`
> ...


Note that you can take as much time as you want for the last step. You might miss out on some cool new features, but otherwise there's no penalty for leaving your zpool at an older version.


----------



## Mjölnir (Mar 29, 2021)

`bectl create previous`
Perform freebsd-update(8) (on _default_ boot environment)... Of course, the chroot(8) method's advantage is reduced interrupt of service
EDIT or use `freebsd-update -b <mountpoint>` /EDIT
reboot(8), run into issues, decide to rollback
"Rollback":

`bectl activate previous`
`reboot`
`bectl destroy default`
`bectl rename previous default`
Conclusio: `freebsd-update rollback` is only for UFS, and then only if you don't have 2 root filesystem partitions to switch between (e.g. IIRC nanobsd(8) does that by default).


----------



## zirias@ (Mar 29, 2021)

FWIW, if you install from source, you can fully install a new release without the need for chroot:

```
bectl create 13rc4
bectl mount 13rc4
cd /usr/src
make DESTDIR=/tmp/be-XXXX installkernel
make DESTDIR=/tmp/be-XXXX installworld
make DESTDIR=/tmp/be-XXXX BATCH_DELETE_OLD_FILES=yes delete-old
bectl umount 13rc4
bectl activate 13rc4
```

But then, having to use chroot isn't _that_ bad either. When upgrading to a new release, I'd always prepare it in the new BE, not in the currently running one, because otherwise it isn't safe to install the userland without booting with the new kernel first.

Edit: Some support for a destination dir in freebsd-update(8) would be nice!


----------



## Mjölnir (Mar 29, 2021)

From RTFM freebsd-update(8)

```
OPTIONS
     The following options are supported:

     -b basedir     Operate on a system mounted at basedir.  (default: /, or
                    as given in the configuration file.)
```
Unfortunately, that option is not mentioned in freebsd-update.conf(5)


----------



## zirias@ (Mar 29, 2021)

Ha! I didn't even look, cause everyone seems to describe the chroot method. Looks like it isn't necessary after all.


----------



## grahamperrin@ (Apr 20, 2021)

The chroot method is appealing for its ability to streamline things, however it's too easy for things to go wrong if the end user follows on-screen instructions.


----------



## freezr (Nov 8, 2021)

Mjölnir said:


> `bectl create previous`
> Perform freebsd-update(8) (on default boot environment)... Of course, the chroot(8) method's advantage is reduced interrupt of service
> EDIT or use `freebsd-update -b <mountpoint>` /EDIT
> reboot(8), run into issues, decide to rollback
> ...



Thank you very much, this was revealing!

Just a quick question on another "tutorials" I read to use -r as in:

`bectl create -r previous`



> The -r option is the recursive option, it is needed to make sure we get all the relevant datasets.
> source: https://unixsheikh.com/tutorials/migrating-zfs-from-linux-to-freebsd.html



Is it important this detail in your opinion?


----------



## grahamperrin@ (Nov 9, 2021)

I don't fully understand the pros and cons of recursion, in this context, but I do recall a suggestion that it might become a default.

I'm checking in a chat room.


----------



## Selin (Nov 9, 2021)

tgl said:


> Is it important this detail in your opinion?


I think this might be useful when different ZFS volumes are used for the system files - to have snapshots of all these volumes.


----------

