# ZFS i/o error - all block copies unavailable after freebsd-update



## schojo (Jun 14, 2021)

Hi all,
i just did a freebsd-update fetch, freebsd-update install and reboot from 13.0-RELEASE to 13.0-RELEASE-p2 and get greeted with this:





Just yesterday everything was fine, zpool status had all four hard disks ONLINE... what can i do? :/


----------



## covacat (Jun 14, 2021)

can you boot from removable media ?
how old is your efi partition code ? it comes from a 13 install or earlier ?


----------



## schojo (Jun 14, 2021)

covacat said:


> can you boot from removable media ?
> how old is your efi partition code ? it comes from a 13 install or earlier ?


I am making one as we speak so i can at least do SOMETHING on this machine...

Not sure what you mean by efi partition code but this is a machine coming from 12.1 and doing upgrades ever since.


----------



## SirDice (Jun 14, 2021)

schojo said:


> but this is a machine coming from 12.1


Did you upgrade your pool after you upgraded to 13.0? And did you also update your boot partition and/or EFI boot? You might still have an older boot partition that cannot read the new ZFS.


----------



## schojo (Jun 14, 2021)

SirDice said:


> Did you upgrade your pool after you upgraded to 13.0? And did you also update your boot partition and/or EFI boot? You might still have an older boot partition that cannot read the new ZFS.


pretty sure that i did, yes.


----------



## SirDice (Jun 14, 2021)

At least grab a 13.0-RELEASE install media, boot from that and see if you can import the pool.


----------



## schojo (Jun 14, 2021)

SirDice said:


> At least grab a 13.0-RELEASE install media, boot from that and see if you can import the pool.


zpool list is not finding any, but zpool import gives me the message below - should i try to?


----------



## SirDice (Jun 14, 2021)

You probably need to force it as the pool is still registered on the original system (that's what it's telling you). You'll also need to add `-R` or else the mountpoints are going to clobber all over the boot disk's mountpoints.


----------



## covacat (Jun 14, 2021)

install the 13.x loader.efi in you efi boot partition
then you'll probably be able to boot from your zfs pool
mount -t msdosfs /dev/ada0px /mnt
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi


----------



## SirDice (Jun 14, 2021)

Yeah, I suspect you only updated the bootcode in the freebsd-boot partition, and forgot to update the EFI bootcode.


----------



## schojo (Jun 14, 2021)

`zpool import -fR bootpool` does something for a second (including hdd led flickering) and then that's it. zpool list and zpool status are empty still. should've noted this before but the main pool is geli encrypted, not sure about the bootpool - this was an install-and-forget setup and always worked without any problems


----------



## schojo (Jun 14, 2021)

covacat said:


> install the 13.x loader.efi in you efi boot partition
> then you'll probably be able to boot from your zfs pool
> mount -t msdosfs /dev/ada0px /mnt
> cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi


that would work - in theory. i have 4 harddisks so i need to copy that to ada0p1, ada1p1....ada3p1 - correct?
The problem is this - "no space left on device". That  doesn't sound like fun at all:


----------



## Emrion (Jun 14, 2021)

I think that bootpool means this is a MBR installation with zfs on root.
Give us the result of `gpart show`.


----------



## schojo (Jun 14, 2021)

Emrion said:


> bootpool means this is a MBR installation with zfs on root. I think there is no EFI at all.
> Give us the result of `gpart show`.


----------



## covacat (Jun 14, 2021)

try loader_simp instead of loader
that might fit


----------



## Emrion (Jun 14, 2021)

Strange for me. It's maybe because you main pool is encrypted.
You can still put the efi loader. See here: https://forums.freebsd.org/threads/update-of-the-bootcodes-for-a-gpt-scheme.80163/
Look inside the "spoiler".


----------



## schojo (Jun 14, 2021)

covacat said:


> try loader_simp instead of loader
> that might fit


coming back to my question here, copy to all four adaXp1 ?


----------



## schojo (Jun 14, 2021)

covacat said:


> try loader_simp instead of loader
> that might fit


Okay done, now i'm here:


----------



## Emrion (Jun 14, 2021)

Yes, all the drives, it's more secure. it's indicated in my howto.


----------



## SirDice (Jun 14, 2021)

Make sure you have `zfs_load="YES"` in /boot/loader.conf.


----------



## covacat (Jun 14, 2021)

schojo said:


> coming back to my question here, copy to all four adaXp1 ?


all the drives you can boot from. if you can boot from all, then all


----------



## schojo (Jun 14, 2021)

SirDice said:


> Make sure you have `zfs_load="YES"` in /boot/loader.conf.


I'm kinda clueless right now on how i can access my /boot/loader.conf right now - any leads on that?
Also, why would it disappear from there?


----------



## covacat (Jun 14, 2021)

can you drop to loader prompt ?
if you can type load zfs
then boot

docs are not very clear but it seems that simple loader cant do with xxx_load="YES"


----------



## SirDice (Jun 14, 2021)

If I recall correctly you need to load the kernel first. But yes, drop to the loader prompt, load the kernel, load zfs, then `boot`. 


```
load /boot/kernel/kernel
load /boot/kernel/zfs.ko
boot
```
Once the system is booted you can easily edit /boot/loader.conf.


----------



## schojo (Jun 14, 2021)

SirDice said:


> If I recall correctly you need to load the kernel first. But yes, drop to the loader prompt, load the kernel, load zfs, then `boot`.
> 
> 
> ```
> ...


Well, that got me somewhere ... it "works" - i can see lots of dmesg afterwards but then crashes at the end and reboots. Too fast too read sadly :/


----------



## SirDice (Jun 14, 2021)

Try `boot -s` instead, that should boot it in single user mode. Then you can at least fix loader.conf. That will make the booting automatic again.


----------



## schojo (Jun 14, 2021)

SirDice said:


> Try `boot -s` instead, that should boot it in single user mode. Then you can at least fix loader.conf. That will make the booting automatic again.


Still the same, but made a video and got the screen before it crashes:


----------



## SirDice (Jun 14, 2021)

I've seen that 'unlink /dev/dev' before. Can't remember what caused that though. And it's panicking because it can't find init(8). That probably has to do with the fact you have a bootpool and an encrypted zroot. Your loader.conf probably doesn't have the encryption keys set up, so it fails to load zroot which then causes it to panic(9).


----------



## covacat (Jun 14, 2021)

is your root file system on the same pool as /boot ?


----------



## schojo (Jun 14, 2021)

SirDice said:


> I've seen that 'unlink /dev/dev' before. Can't remember what caused that though. And it's panicking because it can't find init(8). That probably has to do with the fact you have a bootpool and an encrypted zroot. Your loader.conf probably doesn't have the encryption keys set up, so it fails to load zroot which then causes it to panic(9).


Thanks for the explanation. Might be connected to what covacat posted just below then?


covacat said:


> is your root file system on the same pool as /boot ?


I honestly don't remember and i regret not being informed about it. I installed it back in the days and it always just worked. I know that i have two zpools, bootpool and zroot - so that might answer your question already? It was a standard install of 12.0, didn't change anything myself...


----------



## covacat (Jun 14, 2021)

if bootpool only contains /boot and everything else is on zroot thats expected.
also if your zroot is encrypted where are the keys stored or you enter a passphrase at every boot ?
you probably have to load geli to at the loader prompt and init it


----------



## covacat (Jun 14, 2021)

if you have a swap partition next to the efi one probably its better to kill them both and create a larger efi partition and a smaller swap
and put the full efi loader back


----------



## schojo (Jun 14, 2021)

covacat said:


> if bootpool only contains /boot and everything else is on zroot thats expected.
> also if your zroot is encrypted where are the keys stored or you enter a passphrase at every boot ?
> you probably have to load geli to at the loader prompt and init it


Yep, i enter it each time.



covacat said:


> if you have a swap partition next to the efi one probably its better to kill them both and create a larger efi partition and a smaller swap
> and put the full efi loader back


I thought partition number 2 was too small and not efi (partition 1)? there's 492k free after freebsd-boot so maybe i could use those for the bigger loader? Or use the guide that Emrion provided here in the Spoiler: https://forums.freebsd.org/threads/update-of-the-bootcodes-for-a-gpt-scheme.80163/
...which i honestly don't quite understand.


----------



## SirDice (Jun 14, 2021)

Use newfs_msdos(8) to format the efi partition. The 200MB should be big enough. As this is from an older 12.x installation it probably copied loader.efifat to it. Which causes it to have a really small formatted space. Once it's been properly formatted to use the whole 200MB space then you can copy the "full" loader.efi to it.

No need to move or remove partitions to make "more" space available. It's already there, just badly formatted.


----------



## covacat (Jun 14, 2021)

yes, do what SirDice said. I missed the output of gpart in a previous post.
Alternatively switch to csm boot mode and update freebad-boot content (in reverese order)


----------



## Emrion (Jun 14, 2021)

schojo said:


> Yep, i enter it each time.
> 
> 
> I thought partition number 2 was too small and not efi (partition 1)? there's 492k free after freebsd-boot so maybe i could use those for the bigger loader? Or use the guide that Emrion provided here in the Spoiler: https://forums.freebsd.org/threads/update-of-the-bootcodes-for-a-gpt-scheme.80163/
> ...which i honestly don't quite understand.


It's very simple. You reformat your partition in FAT16, create the directory /efi/boot/ and copy loader.efi inside. That works, I used this trick on several machines.

I think you need the full loader.efi to boot your system. But you can also try to boot in legacy BIOS mode (CSM), it should work.


----------



## schojo (Jun 14, 2021)

Thank you so far - will try tomorrow and report back!


----------



## schojo (Jul 14, 2021)

So, "tomorrow" is basically a month. Anyway, life happened and such.
I just went back to get this thing running again. Tried the reformat for ada0p1, copied loader.efi back after creating the dir and booted from ada0. Voila, was greeted by a "GELI Passphrase:"
Entered it and i am in my system again. Thanks to everybody involved, that worked like a breeze! Will now reformat the other 3 and copy loader.efi there as well so everything is bootable in case of emergency.


----------

