# FreeBSD 13 UEFI boot not realy redundant ?



## jcmichot (Oct 6, 2021)

In reference of https://reviews.freebsd.org/D28897

I do not understand this comment:
"For redundant ZFS-root systems, space is reserved for an ESP on all root-pool disks, but only one is initialized and mounted"

For example in simple case with 2 disk in mirror, only the first can boot, if the first disk die the server can't boot and 
external device (disk/stick/...) must be pluged to boot and restore the content of EFI partition. (same for raidz with all disks).

in 12.2l EFI partitions has been setup during the installation for ALL DISKs.

Anyway mount the EFI partition is a bad idea because it's physically linked to a specific disk.

It seem i'm back in time with Linux/Grub problems...


----------



## _martin (Oct 6, 2021)

I think it makes more sense to ask there than here.
grub2 did solve lots of problems, I'm not sure what you referring to. 

I don't know how installer does it, didn't use installer to install FreeBSD for some time now. My rpool (root pool) is mirror pool with the GPT partitions. I do keep legacy boot partition just in case, I've ESP partitions on both disks. Updating EFI contents means manually updating the other (with raidz I'd have to do this for all disks there). 
In case of full disk failure I'm able to boot off from the other disk, be it legacy or UEFI way. 

I'm assuming it's the automation of this process that is an issue there.


----------



## mer (Oct 6, 2021)

My understanding:
If you have a mirror for "root on ZFS" boot, you have 2 physical devices that are able to boot.  
They both have (when looking at  gpart information) a freebsd-boot or efi partition, a freebsd-zfs partition and maybe freebsd-swap partition.  Lets say you have ada0 and ada1.
When the system boots, it can use either of the physical freebsd-boot/efi to actually boot the system, both of them should wind up mounting root from the mirror (probably on the same device), by the time you get to single user, ZFS (loaded from loader) has mounted the mirror pair of devices.
efi or freebsd-boot doesn't matter, you have the same behavior.

The important point is:
The ZFS mirror is on the partitions tagged with freebsd-zfs.
ada0p1 is an efi partition, ada0p2 is an efi partition.  They are not part of the ZFS mirror, they are separate things.  If you need to update boot code, you need to do it to both:
mount ada0p1 /efi0
mount ada1p1 /efi1

Has always been this way.

What you could do during installation:
Create a geom mirror with ada0 and ada1 before you do anything
Create the partitions on the geom mirror.  That means everything is mirrored BUT you cannot create a ZFS mirror because you have already used geom to create a mirror of the raw devices.

I personally don't think of it as a big deal.  Once the boot partition (efi or freebsd-boot) is created and configured, it rarely changes.  If you need to update the boot code on it, all you do is up arrow or cut and paste and change one character.  Letting ZFS control the mirror is more valuable to me than worrying about having to type in the same command twice for different devices (heck sounds like a script to me).

Above is my opinion, take it for what it cost you


----------



## jcmichot (Oct 7, 2021)

mer said:


> My understanding:
> If you have a mirror for "root on ZFS" boot, you have 2 physical devices that are able to boot.
> They both have (when looking at  gpart information) a freebsd-boot or efi partition, a freebsd-zfs partition and maybe freebsd-swap partition.  Lets say you have ada0 and ada1.
> When the system boots, it can use either of the physical freebsd-boot/efi to actually boot the system, both of them should wind up mounting root from the mirror (probably on the same device), by the time you get to single user, ZFS (loaded from loader) has mounted the mirror pair of devices.
> efi or freebsd-boot doesn't matter, you have the same behavior.



My understanding is that your previous quote is true with standard BIOS boot process but not with EFI boot process, at least with my test on DELL PE server.



mer said:


> The important point is:
> The ZFS mirror is on the partitions tagged with freebsd-zfs.
> ada0p1 is an efi partition, ada0p2 is an efi partition.  They are not part of the ZFS mirror, they are separate things.  If you need to update boot code, you need to do it to both:
> mount ada0p1 /efi0
> ...



right but with 12.2 the install process will setup the content of /efi0 AND /efi1 with 13 install there's ONLY content in /efi0 and /efi1 is empty.



mer said:


> What you could do during installation:
> Create a geom mirror with ada0 and ada1 before you do anything
> Create the partitions on the geom mirror.  That means everything is mirrored BUT you cannot create a ZFS mirror because you have already used geom to create a mirror of the raw devices.
> 
> ...



;-)

I'm using geom for at least 15 years, but zfs mirror is much more efficient because it only mirror useful data and not a whole disk/partition when you rebuild large disk/partition it's a significant change.

My experience is that actually install a FreeBSD 13/zfs with BIOS boot produce a reliable server in case of first disk die.
That's not the case if you install with EFI boot and it was ok with FreeBSD 12. Yes i can easily manually fix the problem but only because i found it.

JC
old grumpy 30 years experience sysadmin ;-)


----------



## Argentum (Oct 7, 2021)

jcmichot said:


> For example in simple case with 2 disk in mirror, only the first can boot, if the first disk die the server can't boot and
> external device (disk/stick/...) must be pluged to boot and restore the content of EFI partition. (same for raidz with all disks).


If you have ZFS mirror, you can boot from both drives (I have such system). Also you can remove one or another drive from mirror and boot a removed drive on another system. For that, it is a good idea to reserve efi partition on both drives and copy the loader to both of these partitions. /boot/loader.efi


----------



## jbo (Oct 7, 2021)

jcmichot said:


> My experience is that actually install a FreeBSD 13/zfs with BIOS boot produce a reliable server in case of first disk die.
> That's not the case if you install with EFI boot and it was ok with FreeBSD 12. Yes i can easily manually fix the problem but only because i found it.
> 
> JC
> old grumpy 30 years experience sysadmin ;-)


I don't have 30 years of experience (I am mentioning that because you seem to care a lot about that) but I run quite a few FreeBSD servers. Except for three of them, all boot from a ZFS two-way-mirror. Those servers are perfectly capable of booting when one disk dies (last time had this happen is not so long ago).

As others mentioned here before, when updating the bootloader you have to do so on both ESPs to ensure that they "remain equivalent".



> "For redundant ZFS-root systems, space is reserved for an ESP on all root-pool disks, but only one is initialized and mounted"



Only reading this sentence I´d argue that they talk about the boot process. Event if you have multiple ESPs available (because redundancy), only one will be mounted. Which one does not matter as you'll certainly make sure to update the bootloader on all ESPs when performing such an upgrade.


----------



## Argentum (Oct 7, 2021)

jbodenmann said:


> I don't have 30 years of experience (I am mentioning that because you seem to care a lot about that) but I run quite a few FreeBSD servers. Except for three of them, all boot from a ZFS two-way-mirror. Those servers are perfectly capable of booting when one disk dies (last time had this happen is not so long ago).


*Agree*! Also EFI loader does not need to be on the drive you are booting from. For example, when I have 2 way mirror and separate SSD with L2ARC, I can put an extra copy of loader on that SSD and boot the mirror (or just one drive when the other is removed) with that loader. FreeBSD EFI loader has a stand-alone ZFS implementation in it and it boots the first bootable storage pool it finds in the system.


----------



## jcmichot (Oct 7, 2021)

jbodenmann said:


> I don't have 30 years of experience (I am mentioning that because you seem to care a lot about that) but I run quite a few FreeBSD servers. Except for three of them, all boot from a ZFS two-way-mirror. Those servers are perfectly capable of booting when one disk dies (last time had this happen is not so long ago).



sure i care of old grumpy ;-)

I will do more test next week, but i point the missing EFI loader (efi partition empty except on first disk) after a fresh install in 13 and that was not the case in 12.


----------



## mer (Oct 7, 2021)

I think the OPs point is that 12.x installer set both up correctly, whereas the 13.x installer does not if you set up to use EFI instead of BIOS.
It also sounds like he knows how to correct the issue, just thinks he shouldn't have to.

jcmichot is that a correct understanding?
If so, he's not wrong.


----------



## Argentum (Oct 7, 2021)

jcmichot said:


> I will do more test next week, but i point the missing EFI loader (efi partition empty except on first disk) after a fresh install in 13 and that was not the case in 12.


I see. Personally, I must confess, that I have never used the 13.x installer. Usually only clone and uprade previous systems.
But this is probably not a big problem. It is good that at least it creates empty efi partitions.

And if you say so that the partition is created empty, I think it is a bug and should be reported.


----------



## jcmichot (Oct 7, 2021)

mer said:


> jcmichot is that a correct understanding?
> If so, he's not wrong.



yes


----------



## agneau (Oct 7, 2021)

mer said:


> I think the OPs point is that 12.x installer set both up correctly, whereas the 13.x installer does not if you set up to use EFI instead of BIOS.
> It also sounds like he knows how to correct the issue, just thinks he shouldn't have to.
> 
> jcmichot is that a correct understanding?
> If so, he's not wrong.


And one more issue is the /boot/efi permanently mounted from the first disk of the pool. If that physical disk fails, you have one mounted filesystem from a failed disk. This may jeopardize the stability of the system. One should never mount permanently a filesystem from a non-raid device on a server build for redundancy.


----------



## astyle (Oct 7, 2021)

The only real difference I observed between UEFI bootloader and non-UEFI is that UEFI bootloader allows you to use nice shiny graphics, while the non-UEFI version has to stick to rather large text. I'm running 13-RELEASE, and never had a problem with either. Maybe it helps that I use SSD's...


----------



## agneau (Oct 7, 2021)

Argentum said:


> I see. Personally, I must confess, that I have never used the 13.x installer. Usually only clone and uprade previous systems.
> But this is probably not a big problem. It is good that at least it creates empty efi partitions.
> 
> And if you say so that the partition is created empty, I think it is a bug and should be reported.


I just send the following pr https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258987


----------



## Julf (Oct 15, 2021)

Is there a manual fix for now? Just format the EFI partitions on the other disks as msdos file systems and copy everything from the EFI partition on the first disk?


----------



## mer (Oct 15, 2021)

Julf said:


> Is there a manual fix for now? Just format the EFI partitions on the other disks as msdos file systems and copy everything from the EFI partition on the first disk?


With the caveat that I have not personally done this, yes, that should work.


----------



## Julf (Oct 15, 2021)

mer said:


> With the caveat that I have not personally done this, yes, that should work.



Thanks! Is gpart bootcode also needed (for the other disks)?


----------



## mer (Oct 15, 2021)

Julf said:


> Thanks! Is gpart bootcode also needed (for the other disks)?


I do not know the answer to that one, someone with more UEFI experience should answer, but I think not.  My understanding is the gpart bootcode is how the system gets from BIOS to booting UFS or ZFS.  UEFI I think looks for efi type of partitions and then the data in them is how it boots (basically replaces BIOS to gpart bootcode/loader).

I could be completely wrong, so maybe someone like astyle or Argentum can better answer.


----------



## Emrion (Oct 15, 2021)

No `gpart bootcode` for UEFI. You have just to copy a file in the efi partition (at the proper place).

See https://forums.freebsd.org/threads/update-of-the-bootcodes-for-a-gpt-scheme.80163/


----------

