# root on ZFS with separate /boot



## luqasz (May 13, 2018)

Hi

I am new to freeBSD and I'd like to use it mainly for ZFS. I've bought HPE MicroServer Gen8 with two 1TB disks. My plan is to use microSD card for /boot so that bios always have one place to boot from. I do this trick when I install mdadm on linux because mdadm does not replicate whole disk (including MBR record). I've read this tutorial, but I am lost as install requires so many steps that I do not understand. Does freeBSD require separate volumes for /usr, /tmp etc. ?

Maybe I am wrong with a separate /boot. Please advise.


----------



## ShelLuser (May 13, 2018)

First issue: what are you trying to accomplish by all this?

See: your reasoning doesn't add up to me. "_Always having one place to boot from_"? So why not just use the HD's? Even SD cards can fail you know....

My advice: don't overcomplicate stuff, its not worth it. Why assume that the boot will succeed when the userland is up in pieces? This setup won't give you any direct advantages. In fact, I dare argue it's going to be highly counter productive, especially if you need to upgrade the system (but this does depend on context).


----------



## luqasz (May 13, 2018)

> First issue: what are you trying to accomplish by all this?


I have one scenario in mind.

Expand zfs pool with additional two 1 TB drives.
My first two drives go bad.
How do I replicate boot / mbr / uefi to other drives ?



> Even SD cards can fail you know....


Much rare that regular spinning drives.


----------



## ucomp (May 14, 2018)

luqasz said:
			
		

> Hi
> 
> I've read this tutorial, but I am lost as install requires so many steps that I do not understand.
> ....... How do I replicate boot / mbr / uefi to other drives ?


Forget those tutorials, read this one  ;-) : https://forums.freebsd.org/threads/clone-a-zfs-disk.57970/#post-368155
you will simply install the boot-OS from CD/DVD and set the boot-filesystem to wherever you want


----------



## SirDice (May 14, 2018)

luqasz said:


> I have one scenario in mind.
> 
> Expand zfs pool with additional two 1 TB drives.
> My first two drives go bad.


1) Can be done "on-the-fly". You don't even have to unmount anything. 
2) Then you're screwed and having /boot on an SD card isn't going to make it easier to fix things, /boot only contains the kernel. And you can't do anything with just the kernel.


----------



## luqasz (May 14, 2018)

This is my gpart list output:

```
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: a930306e-5493-11e8-b8ac-94188238e61c
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: gptboot0
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: ada0p2
   Mediasize: 1073741824 (1.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e0
   rawuuid: a9703b8d-5493-11e8-b8ac-94188238e61c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap0
   length: 1073741824
   offset: 1048576
   type: freebsd-swap
   index: 2
   end: 2099199
   start: 2048
3. Name: ada0p3
   Mediasize: 999129350144 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: a9928e4c-5493-11e8-b8ac-94188238e61c
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: zfs0
   length: 999129350144
   offset: 1074790400
   type: freebsd-zfs
   index: 3
   end: 1953523711
   start: 2099200
Consumers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e3

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: aa3327d4-5493-11e8-b8ac-94188238e61c
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: gptboot1
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: ada1p2
   Mediasize: 1073741824 (1.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e0
   rawuuid: aa7ae983-5493-11e8-b8ac-94188238e61c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap1
   length: 1073741824
   offset: 1048576
   type: freebsd-swap
   index: 2
   end: 2099199
   start: 2048
3. Name: ada1p3
   Mediasize: 999129350144 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: aaa339fe-5493-11e8-b8ac-94188238e61c
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: zfs1
   length: 999129350144
   offset: 1074790400
   type: freebsd-zfs
   index: 3
   end: 1953523711
   start: 2099200
Consumers:
1. Name: ada1
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e3

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 976771021
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 187695104 (179M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r0w0e0
   rawuuid: 5fed3ba0-8ca4-4e4e-9c48-5781b5122664
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: EFI
   length: 187695104
   offset: 1048576
   type: efi
   index: 1
   end: 368639
   start: 2048
2. Name: ada2p2
   Mediasize: 305525686272 (285G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 188743680
   Mode: r0w0e0
   rawuuid: e5413ccc-b4bb-4d64-954c-ca1b66fb7a52
   rawtype: ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
   label: primary
   length: 305525686272
   offset: 188743680
   type: ms-basic-data
   index: 2
   end: 597098495
   start: 368640
3. Name: ada2p3
   Mediasize: 194391310336 (181G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 771751936
   Mode: r0w0e0
   rawuuid: fac1803b-3d80-40c5-bf39-c47972b1836f
   rawtype: 0fc63daf-8483-4772-8e79-3d69d8477de4
   label: root
   length: 194391310336
   offset: 305714429952
   type: linux-data
   index: 3
   end: 976769023
   start: 597098496
Consumers:
1. Name: ada2
   Mediasize: 500106780160 (466G)
   Sectorsize: 512
   Mode: r0w0e0
```

zpool status:

```
pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Fri May 11 22:24:56 2018
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0
        ada1p3  ONLINE       0     0     0

errors: No known data errors
```

As you can see I have zfs on a partition. Not whole disk. When I add two more disks and expand zfs, I still need to clone somehow freebsd-boot partitions. Maybe I am missing something about boot. Maybe I just need freebsd-boot to be on SD card.


----------



## SirDice (May 14, 2018)

luqasz said:


> When I add two more disks and expand zfs, I still need to clone somehow freebsd-boot partitions.


No, because you will never boot from them. Suppose you added another mirror set to this existing pool, only the first two drives could potentially be used to boot the system (that's ada0 or ada1). That's why the boot is copied between those two disks. Additional disks would never be accessed to boot from. If the first two disks (ada0 and ada1) of the mirror set die, at the same time, the whole pool will be destroyed and unavailable any way. Hence trying to boot from the additional disks would be futile. 

Thus additional mirror sets can be added "whole", no need for partitions. But even if you "cloned" the boot partitions and created a ZFS partition, you can still add them "on-the-fly". There's really no need to shutdown or unmount anything. That's what makes ZFS so incredibly useful for storage.



luqasz said:


> Maybe I just need freebsd-boot to be on SD card.


That will do even less. The only thing the code in the freebsd-boot partition does is to find the filesystem to boot from and load zfsloader(8) from it. It's not going to do anything if those filesystems are destroyed due to a (catastrophic) failure. And there won't be a lot of useful tools to fix the situation at that stage. So it's rather pointless and only complicates things.


----------



## luqasz (May 14, 2018)

So in my case I just need to create partitions and add `ada2p3` and `ada3p3` as a mirror set into my pool ? Also add swap partition in fstab. Will `freebsd-boot` partitions be replicated somehow ?


----------



## ucomp (May 14, 2018)

luqasz said:


> ..... create partitions and add `ada2p3` and `ada3p3` as a mirror set into my pool ?.......



Not a  surprise that you're completely confused about ZFS after reading those tutorials  you have nothing to do with partitions .... in ZFS you can create volumes not partitions  . You will not add mirrors to your root pool, but you can create/add as much as mirrors as you have disks to other pools. the next thing is: snapshots. You can e.g create a snapshot from your current root pool. Than you can e.g install freebsd to another mirrored pool, after that you can OVERWRITE your new root pool by sending the old root pool to the newly installed disk. (boot your system NOT from your new installation, you cannot override your booted root pool).And then your new disk is BOOTABLE, completely new disks, bootable with your old root pool. You even can install freebsd( or Sun OS  ) to a remote ZFS-volume and boot it e.g. via fibrechannel from another machine or via kvm or so .... you can do a lot... zfs makes portability of UnixOSes easy like Docker does in linux.


----------



## giahung1997 (May 15, 2018)

Don't let it confuses you because of your previous experiences with the Linux tools and Linux styles. If you switch the FBSD, learn to use FBSD tools and FBSD styles, forget all your previous Linux ways.


----------



## SirDice (May 15, 2018)

luqasz said:


> So in my case I just need to create partitions and add `ada2p3` and `ada3p3` as a mirror set into my pool ?


No, you don't create _any_ partitions on the additional disks. You add the disks "whole", i.e. without partitions. 



> Will `freebsd-boot` partitions be replicated somehow ?


No, never. Besides that you don't need it on the additional disks.


----------



## ucomp (May 15, 2018)

giahung1997 said:


> ...... forget .... Linux .....


maybe a  good idea , I don't know   lol
but that has nothing to do with zfs because ZFS is also available for Linux .
Could even be a good reason to use freebsd as a zfs-server(to share zfs-volumes) for Linux because zfs is the (bootable)default in fbsd(if you want). The difference with zfs on Linux is that its not the linux-default and not default-bootable. You have to take some effort to make zfs bootable in linux.

Back to topic ;-)
What the TS is confused by is the difference from HW-RAID ( or SoftRaid) to ZFS.
ZFS has nothing to do with hw-raid, partitions or megacli ...
that's the reason why  zfs-users are always looking for IT-mode-flashable-HBAs to disable all
HW-raid-features for the server-backplanes.
Another way is to connect the drives to motherboard-ports or other HBAs (SAS-JBODs e.g. )

ZFS....


SirDice said:


> 1) Can be done "on-the-fly".


that's a nice description , its a flying filesystem  , completely independent of waiting 1 million hours until megacli has initialized a disk ;-)
"flying" means e.g. : zfs is 'platform-independent'( as long as zfs is implemented), you can create a zfs-volume under freebsd , pull the disk out, put it in a SunOS or linux-machine and  type zpool import( if not detected by default)...


----------



## SirDice (May 15, 2018)

ucomp said:


> that's a nice description


https://en.wikipedia.org/wiki/On_the_fly


----------



## ucomp (May 15, 2018)

SirDice said:


> https://en.wikipedia.org/wiki/On_the_fly    : ..... Switching computer parts on the fly means computer parts are replaced while the computer is still running......


that's why I said its a really nice(really good) description for zfs ,
the difference to HW-Raid is e.g the you cannot switch (HW-raid-) computerparts( detect disks ) if you don't have the exact same HBA  and same OS in the other machine.
ZFS can do it ...
so from now on ZFS means : Zettabyte FLYING System


----------



## luqasz (May 15, 2018)

Yes freebsd is not linux. That is 100% correct. 
What confuses me is the fact that when I swap disks (so that I add them whole e.g. `ada3` and not `ada3p1`) to a situation where I no longer have a disk with `freebsd-boot`, system is not bootable. Does zfs prevent this situation ? `freebsd-boot` is a mandatory thing ? 
Well I do not even have to swap all disks. In bios / uefi you can choose boot oder. Most of them will give you disk model number and with all disk same model, which one has a boot loader and which one does not. 

I am not being paranoid. I had this experience when I had to drive kilometers to a server because bios did not boot from disk. Did not boot because linux mdadm replicates partitions and not boot sectors which is different.


----------



## ucomp (May 15, 2018)

luqasz said:


> ----
> where I no longer have a disk ........ system is not bootable. Does zfs prevent this situation ?


one of the rare things  zfs can't do for you :
you can't boot an operating system when its not installed on an accessible disk ( or Lun or whatever)   
... but let's begin from the start :
your thread title : "root on ZFS with separate /boot" ,
Answer: It's separated by default
 and look this :


SirDice said:


> The only thing the code in the freebsd-boot partition does is to find the filesystem to boot from and load zfsloader(8) from it.




1st thing you do is disabling HW-RAID in Bios on your HP-machine.
As you know: 


luqasz said:


> In bios / uefi you can choose boot oder.


You'll install freebsd (for redundancy) on e.g. mirrored SSDs  of the size of your choice.
You will NOT remove these 2 boot-disks from the system .
And then it comes to  SirDice  - *you will learn to fly * with whatever  your fantasy and zfs can offer you with additional disks, Luns, KVMs, Volumes .     
 please read the zfs docs ( not old tutorials from times where zfs was not in the freebsd- DVD/CD- Installation- disks ) , read this thread again if it comes to situations where you want e.g. to clone a bootable zfs-disk .... have fun !


----------



## luqasz (May 15, 2018)

Aha. So I need to place `freebsd-boot` to a sd card, Set bios / uefi to boot from it, and now I can use whole disk in a zfs pool. Am I right ?


----------



## zirias@ (May 15, 2018)

I still don't get what you expect to gain from a separate boot medium? This might make sense if you're aiming for full-disk encryption -- otherwise, it's just unnecessary hassle


----------



## ucomp (May 15, 2018)

luqasz said:


> Aha. So I need to place `freebsd-boot` to a sd card, Set bios / uefi to boot from it, and now I can use whole disk in a zfs pool. Am I right ?


Normally you don't have to place something, the fbsd-DVD/CD/USB-installer(.iso) will do it for you 
yes you can install freebsd on an sd- card , but a better choice for your machine is to add one  "S" to SD, then its an SSD(better mirrored pool, 2 SSDs)   
But there's no disadvantage to install freebsd on your 1Tb-disks( only slower performance than SSDs), you can create volumes on zfs-pools .


luqasz said:


> now I can use whole disk in a zfs pool. Am I right ?


I`m not sure I understand what you exactly mean but you can use every detected disk for ZFS, yes.You can create pools(whole disks),create volumes on that pools( size of your choice, expandable, shrinkable) , 

and again :


SirDice said:


> The only thing the code in the freebsd-boot partition does is to find the filesystem to boot from and load zfsloader(8) from it.


forget the boot-thing but don't remove your boot-disks from the system ;-)
install freebsd wherever you want , don't remove that disks and create volumes or whatever on those disks you want


----------



## sko (May 16, 2018)

luqasz said:


> What confuses me is the fact that when I swap disks (so that I add them whole e.g. `ada3` and not `ada3p1`) to a situation where I no longer have a disk with `freebsd-boot`, system is not bootable.



If there is no disk with a boot loader, the system is not bootable. This has nothing to do with ZFS or FreeBSD...



luqasz said:


> Well I do not even have to swap all disks. In bios / uefi you can choose boot oder. Most of them will give you disk model number and with all disk same model, which one has a boot loader and which one does not.



This was true for legacy boot where you had to specifically tell the BIOS from which disk it should boot. With EFI the BIOS/UEFI probes all disks for a valid EFI partition and boot loaders and offers them directly. This *should* even work if you put several disks with EFI partitions/slices and boot loaders into the system, but on consumer hardware UEFI is usually broken in some way or another, so don't expect it to work as intended/specified by the standard or even as common sense would imply...


If you want to separate user data from the system installation, just put a pair of SSDs in the system, install FreeBSD on these and use the other disks for a separate storage pool. This is usually the way to go for most servers, especially storage/file servers as it greatly simplifies migration of the data (just export the pool and pull out the disks).
Don't use SD-Cards or USB flash drives for R/W installations. They _will_ fail in very short time. Only use them for read-only systems like e.g. hypervisors or special OS variants that have been designed/modified for this use case (e.g. smartOS)


----------

