# Upgrading root pools on FreeBSD 13?



## `Orum (Apr 18, 2021)

After upgrading to FreeBSD 13, I went to enable the fancy new features on my root pool, only to find that the typical `gpart bootcode ...` command/warning was no longer present.  I assume this is still needed, but I wanted to confirm that is correct before running it.  I'm also a bit curious why it was removed, and if the commands should be the same as they were before.


----------



## xtaz (Apr 18, 2021)

You absolutely still need to update the boot loader using the same command. It will fail to boot if you upgrade the root pool without updating the loader.

That warning was part of the original ZFS code from Illumos. 13 has switched to using the ZFS code from OpenZFS instead now, and that warning is no longer present. It could easily be patched back in, but for some reason it hasn't been.


----------



## zirias@ (Apr 18, 2021)

I was always unhappy with the previous warning, as it could easily mislead a user into thinking the example command (for upgrading "legacy" boodcode) would be correct for any system, while upgrading UEFI bootcode works very different…

Still, not having a warning at all seems worse


----------



## mickey (Apr 18, 2021)

Zirias said:


> I was always unhappy with the previous warning, as it could easily mislead a user into thinking the example command (for upgrading "legacy" boodcode) would be correct for any system, while upgrading UEFI bootcode works very different…


The instructions in /usr/src/UPDATING are also quite suboptimal and do not mention any procedures for updating the UEFI bootcode at all:

```
ZFS notes
        ---------
        When upgrading the boot ZFS pool to a new version, always follow
        these two steps:

        1.) recompile and reinstall the ZFS boot loader and boot block
        (this is part of "make buildworld" and "make installworld")

        2.) update the ZFS boot block on your boot drive

        The following example updates the ZFS boot block on the
        freebsd-boot partition of a GPT partitioned drive ada0:
        "gpart bootcode -p /boot/gptzfsboot -i $N ada0"
        The value $N will typically be 1 (if booting from BIOS) or 2 (if
        booting from EFI).

        Non-boot pools do not need these updates.
```
And now with /boot/boot1.efifat gone (which imo is a good thing) the situation doesn't get any better. If you had previously used the efifat image you will probably run into the problem that the filesystem does not have enough space to hold FreeBSD 13's bootloader. Which brings us to newfs_msdos(8) and the choice of FAT filesystem. My understanding is that the UEFI specification *only* mandates support of FAT12 and FAT16 for *removable* drives, which implicates that on a fixed drive it is supposed to be FAT32. Easy enough you'd think, just use `newfs_msdos -F 32 <dev>`. But doing that on one of my machines resulted in newfs_msdos(8) complaining about not having enough clusters for a FAT32 filesystem *bummer*. So next thing was to experiment in dry-run mode with the -b or -c flags to find a useable cluster size. When finally all parameters looked correct and I tried to actually create the new filesystem on the ESP, newfs_msdos(8) spits out some really bad looking complaint about KERN_MAXPHYS, making me doubt if it actually had created the filesystem, but it turned out that despite that message it had done it's job *phew*.


----------



## zirias@ (Apr 18, 2021)

mickey said:


> My understanding is that the UEFI specification *only* mandates support of FAT12 and FAT16 for *removable* drives, which implicates that on a fixed drive it is supposed to be FAT32.


I think this part of the spec is somehow questionable, and in fact didn't come across an UEFI implementation that wouldn't boot fine from a FAT12 ESP even on non-removable media. But it seems such implementations exist out there   (yes, I don't get how this should make sense, anyways…)

So, to be on the safe side, you need a somewhat large ESP. The new default is 200MB. Well, for the usecase of only wanting to boot FreeBSD, which requires bootcode of less than 1MB, this is really excessive. But it seems nobody cares for megabytes any more 

On the general topic: This is just a (rare!) case of bad documentation. Someone should do something about it!


----------



## _martin (Apr 18, 2021)

mickey: It's interesting how UEFI standard defines the use of FAT32 (basically what you've mentioned). But then MS defines the way how FAT16 and FAT32 FSs are created. It depends also on the medium size (i.e. there's no such thing as FAT32 of 128MB size in the MS standard).
But then UEFI standard mentions the FS supported by them is based on FAT32 ; I'm assuming it's then sort of custom FS.


----------



## mickey (Apr 18, 2021)

Zirias said:


> So, to be on the safe side, you need a somewhat large ESP. The new default is 200MB. Well, for the usecase of only wanting to boot FreeBSD, which requires bootcode of less than 1MB, this is really excessive. But it seems nobody cares for megabytes any more


The ESP on this machine is already abundantly large:

```
=>        40  3907029088  ada0  GPT  (1.8T)
          40      786432     1  efi  (384M)
      786472        1024     2  freebsd-boot  (512K)
      787496    16777216     3  freebsd-swap  (8.0G)
    17564712  3889464416     4  freebsd-zfs  (1.8T)
```
But still, doing `newfs_msdos -N -F 32 ada0p1` yields:

```
newfs_msdos: 24563 clusters too few clusters for FAT32, need 65525
```
Reducing the number of sectors per cluster to 8 seems to give an acceptable result.
`newfs_msdos -N -F 32 -c 8 ada0p1`

```
/dev/ada0p1: 784864 sectors in 98108 FAT32 clusters (4096 bytes/cluster)
BytesPerSec=512 SecPerClust=8 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 Heads=16 HiddenSecs=0 HugeSectors=786432 FATsecs=767 RootCluster=2 FSInfo=1 Backup=2
```
But when you try to actually create the filesystem, this is what happens:
`newfs_msdos -F 32 -c 8 ada0p1`

```
/dev/ada0p1: 784864 sectors in 98108 FAT32 clusters (4096 bytes/cluster)
BytesPerSec=512 SecPerClust=8 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 Heads=16 HiddenSecs=0 HugeSectors=786432 FATsecs=767 RootCluster=2 FSInfo=1 Backup=2
newfs_msdos: sysctl: KERN_MAXPHYS, using 1048576: Cannot allocate memory
```
That doesn't look right, nevertheless the filesystem has been created:
`mount -t msdosfs /dev/ada0p1 /mnt
df -h /mnt`

```
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada0p1    383M     12K    383M     0%    /mnt
```
So the UEFI bootloader can be installed:
`mkdir -p /mnt/EFI/Boot && install /boot/loader.efi /mnt/EFI/Boot/bootx64.efi`


----------



## `Orum (Apr 19, 2021)

mickey said:


> And now with /boot/boot1.efifat gone (which imo is a good thing) the situation doesn't get any better.


Most of my systems are using the legacy boot, but a few are using EFI.  Somewhat curious why /boot/boot1.efifat was removed, but it sounds like we have to the equivalent of it manually?  I'm not a big fan of that, as there's a lot more potential for things to go wrong.


----------



## zirias@ (Apr 19, 2021)

`Orum said:


> Most of my systems are using the legacy boot, but a few are using EFI.  Somewhat curious why /boot/boot1.efifat was removed, but it sounds like we have to the equivalent of it manually?  I'm not a big fan of that, as there's a lot more potential for things to go wrong.


Multiple reasons:

boot1.efifat was meant to overwrite the whole ESP. That's not a good idea if you have anything else than just a FreeBSD bootloader in there.
boot1.efifat contained boot1.efi as the EFI bootloader, which is deprecated. With EFI, there's no need to boot in multiple stages, you just use loader.efi directly. boot1.efi misses a few features, like for example booting from GELI encrypted pools.
As mentioned above, a (very) few machines actually require the ESP to be formatted with FAT32. For that, it must be large enough. Providing such a large image wouldn't be a good idea.
All in all, there's a lot less that can go wrong now. If all you want to do is to boot FreeBSD, all you have to do is to copy loader.efi on the ESP as efi/boot/bootx64.efi.


----------



## free-and-bsd (Apr 19, 2021)

Had to figure this out, as one of my machines just won't boot GRUB period.
So yes, FAT32 requires some unreasonable EFS sizes, then my system didn't even boot from EFS sized 40M. Buggy BIOS, I guess?

On the other hand, `newfs_msdos -F 12` (for FAT12) works perfectly well with EFI size=1~2M. Because EFI sized 800k (per some HOTOs out there) wont' even contain /boot/loader.efi. 
Yet another quick/dirty fix was to mdconfig the installation ISO, then dd-write /dev/md0p1 to my newly created EFI partition sized 1M. Was really tired of booting/rebooting from one installation to the other to make my router system boot...


----------



## Emrion (Apr 19, 2021)

I stumbled on the problem of copying the 13.0-RELEASE loader.efi on an initial installation with 12.0-RELEASE.

Let's look (efi partition is ada0p1):

```
file -s /dev/ada0p1
/dev/ada0p1: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4  ", root entries 512, sectors 1600 (volumes <=32 MB), sectors/FAT 5, sectors/track 63, heads 1, serial number 0xbd4111ee, label: "EFISYS     ", FAT (12 bit), followed by FAT
```

The room was:

```
df -h
(...)
Filesystem                                 Size    Used   Avail Capacity
/dev/ada0p1                                780K    480K    300K    62%
```
And the file you need to copy is 896 000 bytes (13.0-RELEASE loader.efi).

But... An initial installation of 12.1-RELEASE actually brings an efi partition of FAT16 type:

```
file -s /dev/ada0p1
/dev/ada0p1: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4  ", sectors/cluster 16, root entries 512, sectors/FAT 100, sectors/track 63, heads 16, sectors 409600  (volumes > 32 MB), serial number 0x41af1af7, label: "EFISYS     ", FAT (16 bit)
```
Here, you have access to the 200 MiB of the partition (almost).

So, from my point of view, the simpliest is to "newfs" the efi partition with FAT16 and then, copy the 13.0-RELEASE efi loader.


----------



## mickey (Apr 19, 2021)

`Orum said:


> Most of my systems are using the legacy boot, but a few are using EFI.  Somewhat curious why /boot/boot1.efifat was removed, but it sounds like we have to the equivalent of it manually?  I'm not a big fan of that, as there's a lot more potential for things to go wrong.


Actually no. The necessity to reformat the ESP is caused by having used a filesystem image smaller than the actual partition size in the past, so you will only have to do this once. Future updates of the boot code will only require to mount the ESP, copy the updated bootloader and then unmount it again when you're done. Not using raw writes to a device file also eliminates the risk of getting the index wrong and thereby ruining an innocent filesystem. For the sake of convenience you might even add an entry for your ESP to /etc/fstab. Depending on the complexity of your boot setup, overwriting the ESP with a raw filesystem image could easily turn out to be  anything from a bad idea to a fatal choice. My desktop machine for example runs FreeBSD and Windows 10 dual boot via rEFInd boot manager, there are also additional EFI tools installed like memtest86, you can imagine that writing the efifat image to the ESP would pretty much ruin my day.


----------



## `Orum (Apr 19, 2021)

mickey said:


> Future updates of the boot code will only require to mount the ESP, copy the updated bootloader and then unmount it again when you're done. ... For the sake of convenience you might even add an entry for your ESP to /etc/fstab.


Is it a problem to leave the ESP mounted?  Seems putting an entry in fstab would automount it (unless explicitly set otherwise).


----------



## mickey (Apr 19, 2021)

`Orum said:


> Is it a problem to leave the ESP mounted?  Seems putting an entry in fstab would automount it (unless explicitly set otherwise).


A problem? No. Cause for concern? Maybe. There is no real reason to leave it mounted when there is no immediate need to perform updates upon it. Just add the _noauto_ flag in fstab(5) and it wont get automatically mounted.


----------



## `Orum (May 2, 2021)

Sorry to bump this, but I finally got around to updating these EFI systems, and now they're left unbootable.  Originally I just tried to copy over the new /boot/loader.efi onto the original ESP, but the partition wasn't large enough for it.  I ended up reformatting the partition as FAT32, and then copying it over like so (pool is a mirrored root pool on ada0 and ada1):

```
# newfs_msdos -A -F 32 -b 512 -L EFI /dev/ada0p1
# newfs_msdos -A -F 32 -b 512 -L EFI /dev/ada1p1
# mount_msdosfs /dev/ada0p1 /mnt/esp0
# mount_msdosfs /dev/ada1p1 /mnt/esp1
# mkdir -p /mnt/esp0/EFI/BOOT
# mkdir -p /mnt/esp1/EFI/BOOT
# cp /boot/loader.efi /mnt/esp0/EFI/BOOT/BOOTX64.EFI
# cp /boot/loader.efi /mnt/esp1/EFI/BOOT/BOOTX64.EFI
```
I verified the files are there, but I still get dropped into an EFI shell when it boots.


----------



## Emrion (May 2, 2021)

`Orum said:


> Sorry to bump this, but I finally got around to updating these EFI systems, and now they're left unbootable.  Originally I just tried to copy over the new /boot/loader.efi onto the original ESP, but the partition wasn't large enough for it.  I ended up reformatting the partition as FAT32, and then copying it over like so (pool is a mirrored root pool on ada0 and ada1):
> 
> ```
> # newfs_msdos -A -F 32 -b 512 -L EFI /dev/ada0p1
> ...


It's maybe the way you formated the esp partition.
I managed to do like this (see the spoiler).


----------



## `Orum (May 2, 2021)

Emrion said:


> It's maybe the way you formated the esp partition.
> I managed to do like this (see the spoiler).


The only difference I see is that you used FAT16 instead of FAT32 (other than the label, which I don't think makes any difference).  I suppose it could be that FAT16 is supported, and FAT32 not, but I thought the latter was supposed to be most compatible?


----------



## mtu (May 2, 2021)

`Orum said:


> The only difference I see is that you used FAT16 instead of FAT32 (other than the label, which I don't think makes any difference).  I suppose it could be that FAT16 is supported, and FAT32 not, but I thought the latter was supposed to be most compatible?


Since FAT16 is much older, you can expect greater compatibility from it. Also, FAT32 might not work properly on very small partitions (under ca. 250MiB).


----------



## mickey (May 2, 2021)

`Orum said:


> Sorry to bump this, but I finally got around to updating these EFI systems, and now they're left unbootable.  Originally I just tried to copy over the new /boot/loader.efi onto the original ESP, but the partition wasn't large enough for it.  I ended up reformatting the partition as FAT32, and then copying it over like so (pool is a mirrored root pool on ada0 and ada1):
> 
> ```
> # newfs_msdos -A -F 32 -b 512 -L EFI /dev/ada0p1
> ...


Not sure why you would use the _-A_ flag, description says it's useful for SD cards? Other than that, your cluster size is equal to the sector size, not sure if that's a good thing. You can use dry run mode (_-N_ flag) to find the largest acceptable cluster size for a FAT32 filesystem of the given size. Start with for example 16 sectors per cluster (_-c 16_), if that's not acceptable use 8 and so on.


----------



## Emrion (May 2, 2021)

`Orum said:


> The only difference I see is that you used FAT16 instead of FAT32 (other than the label, which I don't think makes any difference).  I suppose it could be that FAT16 is supported, and FAT32 not, but I thought the latter was supposed to be most compatible?


As I already said above, I just (more or less) mimicked what a fresh install of 12.1-RELEASE and later do.


----------



## `Orum (May 2, 2021)

The -A was there because I figured it wouldn't hurt on a HDD, and might be useful on an SSD.  In any case, I've tried without the -A, using FAT16, and not specifying any cluster size at all, and the machine still will simply not boot.  Any other ideas to try?


----------



## free-and-bsd (May 2, 2021)

FAT12 works best with small size EFI partitions.


----------



## `Orum (May 2, 2021)

I've given up on trying to get EFI boot working.  I'm just going to change it to be a legacy BIOS boot instead.

*Edit:* After switching to legacy boot, things worked on the first try. I think I'll just convert the few remaining EFI systems to all be legacy.


----------



## free-and-bsd (May 2, 2021)

`Orum said:


> I've given up on trying to get EFI boot working.  I'm just going to change it to be a legacy BIOS boot instead.
> 
> *Edit:* After switching to legacy boot, things worked on the first try. I think I'll just convert the few remaining EFI systems to all be legacy.


Yes, that depends on the particular system, I guess. 
I have a new computer with miniATX mobo, now that one ONLY boots EFI, nothing else. Before that I used GRUB2 and legacy mode everywhere... but here I was _forced_ to use UEFI -- and did so to my satisfaction. What I didn't like, though, was being _forced_ to use EFI because of the buggy BIOS.


----------

