# Non-trivial (or rather trivial) boot question



## tarkhil (Dec 10, 2021)

Hello

I have a server, a bit aged but still able and fit for most of my needs. Trying to speed it up and free two disk hotswap places, I've perusaded server's owner to purchase two NVME PCI cards for boot and ZFS cache.
After installing them, I've (suddenly!) find out that the motherboard cannot boot from PCI NVME, and I'm advised to purchase a new motherboard. And two new CPUs. And new memory. Not right now, sorry.
So I think of the USB stick with loader telling boot2 to load from NVME.
As far as I understand, that should be possible, but I've found nothing on the subject.
Maybe someone has already accomplished something similar?


----------



## covacat (Dec 10, 2021)

if nvme is not visible in bios you need full kernel on the usb stick
and mount root from nvme
that works for sure and then mount the stick on /boot


----------



## balanga (Dec 10, 2021)

How often do you boot this server? TrueNAS, which is based on  FreeBSD boots quite easily from a USB stick, and usually does not need to reboot for a month or two.


----------



## mefizto (Dec 10, 2021)

Hi covacat,

could you please provide some more details, as I was unable to make it work?

Kindest regards,

M


----------



## covacat (Dec 11, 2021)

just download a minimal install image, write it to a memory stick, mount it and edit its loader.conf
vfs.root.mountfrom="zfs:ntank/boot" or
vfs.root.mountfrom="ufs:/dev/something0p1"
then you can add /dev/da0p2 /boot to your fstab


----------



## mefizto (Dec 11, 2021)

Hi covacat,

thank you for the reply, unfortunately I my understanding of the boot process is insufficient, so it does not help.

Greetings all,

based on the forgoing, I started reading on the boot process in the Handbook.  It is rather difficult to connect the description with the different files setting boot variables, but as best understood from the Handbook quote: (emphasis supplied):


> Stage two, boot2, is slightly more sophisticated, and _understands the FreeBSD file system enough to find file_


that this is performed by gptzfsboot(8), _cf_. 


> *gptzfsboot *tries to find all ZFS pools that are composed of BIOS-visible hard disks or partitions on them


and


> The first pool seen during probing is used as a default boot pool.
> 
> The filesystem specified by the *bootfs* property of    the pool is used as a
> default boot filesystem.  If the *bootfs* property is not set, then the
> ...



The gptzfsboot(8) further states:
*     gptzfsboot* looks for ZFS device    labels


> on    all visible disks and in discovered supported partitions for all sup-
> ported partition scheme types.  The search    starts with the    disk from
> which *gptzfsboot* itself was loaded.  Other    disks are probed in BIOS de-
> fined order.


But, it does not appear to work.  Although it appears that the NVMe drive is visible and has a ZFS filesytem on it, gptzfsboot(8) cannot find it.  And thee are no options that could be set to help the process mentioned by the gptzfsboot(8).

In a different thread https://forums.freebsd.org/threads/running-os-off-usb-thumb-drive-or.79080/, Mjölnir asserted:


> IIRC you can boot from one media, and have rootdev="disk2s1a" in loader.conf(5) to switch to another device.


However, as I understand, the lader.conf(5) is on the pool/filesystem that the gptzfsboot(8), cannot find.

Any help would be appreciated.

Kindest regards,

M


----------



## richardtoohey2 (Dec 12, 2021)

I've done something _similar_ recently and was going to ask if I'd missed a better way.

So my main issue was the Dell R220 doesn't allow booting from NVMe.

I'd used USB UEFI boot to install FreeBSD 13.0 to a single NVME, UFS and that went fine.

But when I unplugged the USB ... no way to get the BIOS to boot from the NVME.

I need to dig out my notes as to what exactly I did for my situation, but this page was the main help:






						Load NVMe Driver in UEFI Shell - ntzyz's space
					

Four months ago I brought an HP Z620 workstation to run virtual machines at home with Hyper-V Server...



					ntzyz.io
				




So I've ended up with a USB stick in the internal USB port and this USB stick is the boot device holding a UEFI shell and a startup.nsh script.

It boots from the USB stick, loads the UEFI shell that uses the startup.nsh to boot FreeBSD from the NVME.

In case the linked page disappears from the internet, I've copied the four lines of the script:

```
load -nc fs0:\NvmExpressDxe-64.efi
connect -r
map -u
fs1:\EFI\BOOT\BOOTX64.EFI
```
The UEFI shell doesn't have an NVME driver, so line 1 does that.  The connect and map commands connect & map to the NVMe device, making it available as fs1 (fs0 being the boot USB device).  The fourth line makes it run the EFI code on the NVMe device, and because I'd previously installed FreeBSD on the NVMe, it carried on booting my previous installation.

The reason I was going to post on the forums - what I've done does work, but there are a few quirks (again have to check my notes) - main one that I can recall is that when I use top it shows a battery line as though it thinks I've got it running on a laptop - but it's a Dell R220 server.

My other questions were about how safe it was to boot from a downloaded UEFI shell & NVME driver.

Finally you talk about two NVMe devices - that won't work in my case with the Dell because the BIOS doesn't support bifurcation - so you _might_ have further issues if you are trying to use a card with two NVMe devices on.

I'll dig out my notes so I can explain exactly what I had to do (and where I got the UEFI shell and NVME driver from), but think this approach _might_ work for you as well, depending on the hardware and what exactly you are trying to do.

I've not used the machine yet, just got to the point of it UEFI booting from USB (as an initial stage) and then booting off NVMe - so there might be flaws in the cunning plan that I'm not aware of yet!


----------



## mefizto (Dec 12, 2021)

Hi richardtoohey2,

although your post appears to be directed at the OP, as I have not mentioned two NVMe devices, thank you for posting it.  Unfortunately, the motherboard that I have does not support UEFI, but perhaps it may be adopted.

In a meanwhile I found a post that separates the /boot/ and the rest of the system https://freebsdfoundation.blogspot.com/2015/04/from-trenches-tips-tricks-edition.html.  Looking at hier(7), perhaps this is the way forward, since the entire /boot/ with all the entities supporting the boot process will be on the boot drive, thus solving the problem that the gptzfsboot(8) has no options for affecting the boot process.

Though I remember reading somewhere that it is not wise to place the /boot/ on its own filesystem, the author is a prominent FreeBSD member, so I will experiment with the approach.

Kindest regards,

M


----------



## tarkhil (Dec 14, 2021)

covacat said:


> just download a minimal install image, write it to a memory stick, mount it and edit its loader.conf
> vfs.root.mountfrom="zfs:ntank/boot" or
> vfs.root.mountfrom="ufs:/dev/something0p1"
> then you can add /dev/da0p2 /boot to your fstab


So the idea is to mount /boot from the stick and everyone else from zfs on nvme. And, as far as I recall UNIX mounting process, I'll do just fine while my kernel/modules versions are the same.
And I'll look into mfsbsd scripts because I do like the idea of totally r/o boot stick. Flash without writing access should last forever...


----------



## covacat (Dec 14, 2021)

beware of flash bit rot (after several years)
happened on a box i know
i think if you update it at every freebsd update is enough to keep the data ok








						USB flash data rot
					

I have an old bootable stick. After 8 years data has changed slightly, enough to panic at every boot (privileged instruction fault in kernel mode). Changed mobo/memory/cpu and whatnot until i figured it out. no fsck errors, no surface scan errors the stick is only used for booting, no write/no...




					forums.freebsd.org


----------

