# Upgrade to 12.2, ZFS bootcode, and EFI text console issues



## mdavis (Nov 17, 2020)

After upgrading from FreeBSD 12.1 to 12.2-RELEASE, my machine boots and runs fine, but it no longer displays a full native resolution text console on a 17" monitor. I get the giant default 80x25 display instead. Running "vidcontrol -i mode" returns no information at all.

This is a 2015 vintage BIOS booting into UEFI mode, ZFS (mirror) on root.  Here is my gpart output:


```
# gpart show
=>        40  1953525088  ada0  GPT  (932G)
          40      409600     1  efi  (200M)
      409640        1024     2  freebsd-boot  (512K)
      410664         984        - free -  (492K)
      411648     4194304     3  freebsd-swap  (2.0G)
     4605952  1948917760     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)

=>        40  1953525088  ada1  GPT  (932G)
          40      409600     1  efi  (200M)
      409640        1024     2  freebsd-boot  (512K)
      410664         984        - free -  (492K)
      411648     4194304     3  freebsd-swap  (2.0G)
     4605952  1948917760     4  freebsd-zfs  (929G)
  1953523712        1416        - free -  (708K)
```

As recommended, I performed a ZFS upgrade which also says to update the bootcode, which I did with:


```
# gpart bootcode -p /boot/boot1.efifat -i 1 ada0
# gpart bootcode -p /boot/boot1.efifat -i 1 ada1
```

(I then later discovered here in these forums that boot1.efifat is no longer used. I also read where using zpool upgrade's suggestion of "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot" can render an EFI BIOS unbootable. So what then is the correct procedure for updating the bootcode on a UEFI+ZFS root?)

I'm not sure what the problem with the video is, but suspect it's due to one of the following:

1. The bootcode did not update properly or I botched it with the above commands
2. The newer bootcode is no longer compatible with this hardware (I tried reverting to the older 12.1 boot1.efifat but it made no difference)
3. Something changed in 12.2 that requires additional configurations to get the console video to operate like it used to
4. Something changed in the 12.2 GENERIC kernel that isn't recognizing my hardware video capabilities correctly

One other thing I noticed: when booting, just before the FreeBSD Daemon menu appears, I'm seeing a very brief blast of text.  It was so fast that I had to get a video of it to replay it. It appears to show a normal EFI startup with support for ZFS, UFS, and sees an "EFI Console". I don't recall seeing this burst of text under 12.1. Is this normal, or indicative of a problem?

Appreciate any suggestions.
Thank you


----------



## mickey (Nov 17, 2020)

mdavis said:


> (I then later discovered here in these forums that boot1.efifat is no longer used. I also read where using zpool upgrade's suggestion of "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot" can render an EFI BIOS unbootable. So what then is the correct procedure for updating the bootcode on a UEFI+ZFS root?)


The command `gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i <index> <disk>` is what you would use with a GPT partitioned disk on a BIOS (non UEFI) machine to write the protective MBR (-b /boot/pmbr) to the rootsector of the disk and the partition boot code (-p /boot/gptzfsboot) to the _freebsd-boot_ partition. UEFI booting doesn't use the _freebsd-boot_ partition at all (unless CSM is enabled in your firmware), but rather boots from the EFI system partition (ESP), which is an MSDOS formatted partition and should be the first partition on the disk. So if you use that command, specifying the index of your EFI partition instead of the _freebsd-boot_ partition, yes, that will probably wreck your system by writing something to your ESP that is neither an MSDOS file system, nor bootcode recognizable by EFI.

Your disk layout shows that you have both, an ESP (index 1) *and* a _freebsd-boot_ partition (index 2), which is fine, as it means you could boot that disk in either UEFI or BIOS mode. But that also means your disks have two distinct boot loaders and both should be updated.

To update the _freebsd-boot_ partition, you can use `gpart -p /boot/gptzfsboot -i <index> <disk>`. Just double or better triple check, that you don't get the index wrong, or you will break something. The index should be that of your _freebsd-boot_ partition, so in your case 2. There should generally be no need to rewrite/update the protective MBR.

As for the ESP, according to uefi(8) /boot/boot1.efi is still the correct loader to use. The file /boot/boot1.efifat is just an MSDOS file system image that contains a copy of /boot/boot1.efi under the default location where EFI expects a bootloader if not configured otherwise (/EFI/BOOT/BOOTX64.EFI for amd64). To update the bootloader on the ESP you can write the entire file system image to the ESP using `gpart bootcode -p /boot/boot1.efifat -i <index> <disk>`. Again, double check to get the index right, it should always be 1, as the ESP is supposed to be the first partition on the disk by definition. Alternatively you can mount the ESP to inspect/update it's contents manually, i.e. `mount -t msdosfs /dev/ada0p1 /mnt`.

That said, some time back I too was having problems with the graphics resolution being switched to some low-res mode during boot instead of keeping the native resolution of 1920x1080. In search for a solution I came across some talk on one of the freebsd mailing lists, where someone suggested that /boot/loader.efi could be used directly to boot the system instead of /boot/boot1.efi and also that loader.efi would eventually replace boot1.efi altogether. Unfortunately I did not bookmark it, so I cannot provide links/further specifics. As I use an EFI boot manager (rEFInd) there was no immediate danger from trying it, as I could just have both, boot1.efi and loader.efi installed to try them out, and using loader.efi indeed solved my problem, and the system now happily boots in 1920x1080 resolution, using loader.efi.


mdavis said:


> One other thing I noticed: when booting, just before the FreeBSD Daemon menu appears, I'm seeing a very brief blast of text. It was so fast that I had to get a video of it to replay it. It appears to show a normal EFI startup with support for ZFS, UFS, and sees an "EFI Console". I don't recall seeing this burst of text under 12.1. Is this normal, or indicative of a problem?


This is normal behaviour I guess. I am too seeing it, but as you say, it's gone in an instant.


----------



## SirDice (Nov 17, 2020)

mickey said:


> but rather boots from the EFI system partition (ESP), which is an MSDOS formatted partition and should be the first partition on the disk.





mickey said:


> Again, double check to get the index right, it should always be 1, as the ESP is supposed to be the first partition on the disk by definition.


This is common practice, not a requirement. The installer will automatically create it as the first partition, yes, but the ESP can be put anywhere on disk. There's only a limit when using UEFI-MBR, in that case it has to reside in the first 2.2TB of a disk. This is due to a limitation of MBR, not UEFI.


----------



## mdavis (Nov 17, 2020)

Mickey/SirDice:

Thank you for this detailed and informative response. I'll be attempting to update both the 1 (efi) and 2 (gpt) partitions as directed above and see if this resolves the console resolution regression.

I'll follow up with my results, though I am unsure about how to install an EFI boot manager and selecting loader.efi.  Is that something that can be loaded via removable media, like a flash drive, just in case my BIOS doesn't like it? Any pointers to that is appreciated.

It's wonderful to get proper, accurate steps all in one place for issues that can really wreck your day if you get it wrong.  (Also, thanks for reformatting my original post ... that goes above and beyond for the community here!  I tried using CODE tags but the preview looked odd and decided against.)


----------



## mickey (Nov 17, 2020)

mdavis said:


> I'll follow up with my results, though I am unsure about how to install an EFI boot manager and selecting loader.efi. Is that something that can be loaded via removable media, like a flash drive, just in case my BIOS doesn't like it? Any pointers to that is appreciated.


Generally you wont need a boot manager if you don't intent to boot multiple operating systems, like in my case, where I use it to boot FreeBSD and Windows 10. The default place where the EFI firmware will look for a bootloader on an amd64 system is /EFI/BOOT/BOOTX64.EFI on the ESP. If you have written the /boot/boot1.efifat image to your ESP, then the file /EFI/BOOT/BOOTX64.EFI on the ESP will be a copy of /boot/boot1.efi. To replace it with /boot/loader.efi you could mount the ESP and copy the file, like:

```
# mount -t msdosfs /dev/ada0p1 /mnt
# cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
# umount /mnt
```
This of course imposes the risk of leaving you with a non booting system, should for whatever reason your firmware not like to boot using loader.efi. In that case you would need to boot from a FreeBSD USB memstick or DVD to repair the bootloader on your ESP.

This is where having a bootmanager makes the task at hand a lot less dangerous, as you can have several different bootloaders simultaenously using different names, and if one fails to boot, you just choose the other. If you want to take a closer look, rEFInd can be found here. It has some rather extensive documentation and can also be booted off a USB stick or even CD, so you can experiment without the need to install it to your system.


----------



## mdavis (Nov 17, 2020)

That's exactly what I'll want to try. This box is a dedicated FreeBSD-based router, firewall, email, file, print server for my home network (which also is the home-office network) so I'd rather just keep things vanilla as much as possible. (No multi-OS booting).  But if the 12.2 update is no longer happy with my EFI hardware like 12.1 was, I think the idea of a boot manager on a separate flash drive would be interesting and safe to explore. Thanks for the pointers.


----------



## mdavis (Nov 18, 2020)

Quick update: the steps to update bootcode here were completed successfully (thanks again!), but 12.2 still boots up in low-res text mode.

I have rEFInd installed on a flash drive and can boot the machine into that successfully.  Any tips on how to get it to boot with /boot/loader.efi instead of the default fallback loader? I've done a bunch of searches as well as looked over the extensive (to say the least) rEFInd pages, but the answer is not materializing conveniently before my eyes.

Is there a simple recipe for this?


----------



## SirDice (Nov 18, 2020)

mdavis said:


> but 12.2 still boots up in low-res text mode.


Look in vt(4):

```
hw.vga.textmode
             Set to 1 to use virtual terminals in text mode instead of
             graphics mode.  Features that require graphics mode, like
             loadable fonts, will be disabled.

...
     kern.vt.fb.default_mode
             Set this value to a graphic mode to override the default mode
             picked by the vt backend.  The mode is applied to all output
             connectors.  This is currently only supported by the vt_fb
             backend when it is paired with a KMS video driver.
```


----------



## mickey (Nov 18, 2020)

mdavis said:


> I have rEFInd installed on a flash drive and can boot the machine into that successfully. Any tips on how to get it to boot with /boot/loader.efi instead of the default fallback loader? I've done a bunch of searches as well as looked over the extensive (to say the least) rEFInd pages, but the answer is not materializing conveniently before my eyes.


rEFInd automatically scans the ESP for bootloaders and presents the ones it did find, so there is no need to configure boot entries first. You can just copy /boot/loader.efi to a new directory on your ESP and rEFInd should be able to find it, i.e.:

```
# mount -t msdosfs /dev/ada0p1 /mnt
# mkdir /mnt/EFI/FreeBSD
# cp /boot/loader.efi /mnt/EFI/FreeBSD/
# umount /mnt
```
As this will copy /boot/loader.efi to a separate directory on your ESP without modifying the existing bootloader in any way, there should be no harm from it, but rEFInd should find it there and be able to boot from it.


----------



## mickey (Nov 18, 2020)

Just for the fun of it, I took the opportunity to upgrade rEFInd on my system to the latest version (0.12.0) and retest the effects of using /boot/boot1.efi vs. /boot/loader.efi to boot my system. Other than the _efi_max_resolution_ setting, there is nothing special in /boot/loader.conf:

```
# CPU Microcode update.
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"

# Set boot max graphics resolution.
efi_max_resolution="1080p"

# Set beastie logo.
loader_logo="beastie"

# ZFS support.
zfs_load="YES"
```
rEFInd is configured with the following settings in refind.conf:

```
# GOP text mode: 0 = 80x25 / 1 = 80x50 /  2 = 100x31 / 3 = 240x56
textmode 3

# GOP graphics mode: 0 = 1920x1080
resolution 0

use_graphics_for windows
```

Using /boot/boot1.efi to boot, the FreeBSD boot menu with beastie logo appears substantially larger in a box centered on the screen (that might be due to the DFP). When the kernel starts booting, I get: _VT(efifb): resolution 1024x768_

Using /boot/loader.efi to boot, the FreeBSD boot menu with beastie logo appears in the upper left corner of the screen and is rather small, which is to be expected at the given resolution/text mode. When the kernel starts booting, I get: _VT(efifb): resolution 1920x1080_

This confirms my findings from like 1-2 years ago, that boot1.efi messes with the display resolution, whereas loader.efi does not.


----------



## mdavis (Nov 18, 2020)

mickey said:


> rEFInd automatically scans the ESP for bootloaders and presents the ones it did find, so there is no need to configure boot entries first. You can just copy /boot/loader.efi to a new directory on your ESP and rEFInd should be able to find it, i.e.:
> ...


Neat... if I'm interpreting this properly, this means modifying or reconfiguring the rEFInd flash drive isn't necessary. Instead, I'd be adding a new EFI loader to the FreeBSD boot drive itself in a safe manner where only rEFInd would see it and use it.  (How that would appear in the rEFInd UI after booting the flash drive I have yet to discover).


----------



## mickey (Nov 19, 2020)

mdavis said:


> Neat... if I'm interpreting this properly, this means modifying or reconfiguring the rEFInd flash drive isn't necessary. Instead, I'd be adding a new EFI loader to the FreeBSD boot drive itself in a safe manner where only rEFInd would see it and use it.  (How that would appear in the rEFInd UI after booting the flash drive I have yet to discover).


No configuration of rEFInd is needed for the actual boot menu entries. There are of course a lot of things configurable: graphics mode, text mode, background image, font, icons and whatnot. I even copied the EFI version of memtest86 to my ESP and rEFInd presents it with a nice memory test icon  It also comes with operating system icons for FreeBSD, Linux, Windows and some other OS and automatically chooses an appropriate one. The additional bootloader files on the ESP are not hidden, they are just in places where the EFI firmware doesn't look by default, which is anything other than /EFI/BOOT/BOOTX64.EFI.


----------



## mdavis (Nov 19, 2020)

RESOLVED!

Final summary of what I learned:

* How to properly write bootcode updates to the drive for both ESP and FreeBSD boot partitions (thank you, mickey)

* boot1.efifat is 800K in 12.2, so apparently that's all the space you get in the ESP FAT partition.  Since it includes /EFI/BOOT/BOOTX64.EFI already, that takes up about half that space. So copying loader.efi (which is over 400K) into its own directory resulted in "no space left on device". I gave up on that idea. I do have 200MB reserved for the EFI partition, though.

* SOLUTION: Setting loader.conf values restored the native screen resolution (thank you, SirDice).


```
hw.vga.textmode="1"
kern.vt.fb.default_mode="1600x1200"
```

I'm hoping this 1600x1200 mode will not blow up if I have to put a monitor on the system that can't support that size.

For good measure, I also left this in loader.conf in case the EFI stuff ever works for me like it used to:


```
efi_max_resolution="1600x1200"
```


----------



## mickey (Nov 19, 2020)

mdavis said:


> boot1.efifat is 800K in 12.2, so apparently that's all the space you get in the ESP FAT partition. Since it includes /EFI/BOOT/BOOTX64.EFI already, that takes up about half that space. So copying loader.efi (which is over 400K) into its own directory resulted in "no space left on device". I gave up on that idea. I do have 200MB reserved for the EFI partition, though.


That's an unfortunate side effect of using a file system image to initialize the ESP. The solution would be to format the ESP with a FAT filesystem using newfs_msdos(8) and then copy the bootloader in place. That would make the entire space reserved for the ESP available for use.


mdavis said:


> SOLUTION: Setting loader.conf values restored the native screen resolution (thank you, SirDice).
> 
> Code:
> hw.vga.textmode="1"
> kern.vt.fb.default_mode="1600x1200"


There are some issues with these settings though. Setting _hw.vga.textmode=1_ seems to solve the resolution problem on the surface, but it also means you will not be able to use features like loadable fonts and some others. The _kern.vt.fb.*_ settings will only work in combination with a KMS video driver and will only get active once the kernel is halfway through the boot process already.


mdavis said:


> For good measure, I also left this in loader.conf in case the EFI stuff ever works for me like it used to:
> 
> Code:
> efi_max_resolution="1600x1200"


I can't even say with certainty what the intended use-case of this setting would be, other than not using a resolution higher than this. As you can see from my results, it surely doesn't keep the system from using a lower resolution. The fact I still have it in my loader.conf(5) might well be a remnant of my testing back then.


----------



## mdavis (Nov 19, 2020)

Just getting the full use of the display is fine for my purpose (a giant tmux array of panels for running various monitoring outputs). No fancy fonts needed. I'm content with plain vanilla.

This was an interesting and unexpected exercise for just doing a point release upgrade. Our FreeBSD machines are so capable of many things once they're running. It's unfortunate that getting them to start up properly is still one of the most fragile, complicated, frustrating challenges since the first release almost 30 years ago. Kind of like pull-start engines.


----------



## mickey (Nov 20, 2020)

mdavis said:


> It's unfortunate that getting them to start up properly is still one of the most fragile, complicated, frustrating challenges since the first release almost 30 years ago. Kind of like pull-start engines.


I'd say that using a single loader.efi, GPT partitioning and ZFS compared to having a three stage boot loader, MBR slices with disklabels and UFS is a huge improvement.


----------



## mdavis (Nov 20, 2020)

For sure.  My comment is more about how difficult it seems to get the right information when so much legacy info persists around the net. I've distilled your comments here, which were singularly clear and helpful, and recorded them in my notes for the future. Fortunately, the console resolution issue appears to be a minor regression in the 12.2 update. I filed a bug.


----------

