# [UEFI/GPT] [Dual-Boot] How to install FreeBSD (with ZFS) alongside another OS (sharing the same disk)



## patovm04 (Jun 7, 2020)

*Important notes:*
1) This tutorial assumes you have the OS you want to dual-boot with already installed on your drive, and that you already have freed up some disk space. Essentially, you will be installing FreeBSD with root-on-ZFS on the remaining free space of the disk, instead of using the entire drive as it's done by default by FreeBSD's installer.
2) This method only works for GPT disk layouts using UEFI mode. Legacy BIOS and MBR disks are not supported. As always, make sure to disable Secure Boot and Fast Boot.
3) Here, we use ada0 (disk 1) for everything, and we assume your disk already has an EFI partition, located in partition 1 (ada0p1). Change it according to your setup, if needed. Run `gpart show` to be sure.
4) We will create our ZFS Pool with "Stripe" as the Virtual Device type (vdev).
5) After completing installation, to be able to choose what OS to boot you'll need to either install rEFInd boot manager, or use GRUB (assuming the other OS is a Linux distro). I personally recommend rEFInd, but I won't detail how to install it here. Neither will I explain how to alter the GRUB config. Just gonna show you how the respective entries should look like in each case.
6) Finally, I'm not responsible for any problem or data loss you may experience. This tutorial is not an unsafe procedure if you understand what you're doing (especially in regards to selecting the correct disk where you want to install). Anyway, do it at your own risk!


Well, let's start:
First of all, boot your FreeBSD install USB stick. Proceed installing as usual, until you reach the "Partitioning" stage. Here, choose the "Shell" option, so you can create your swap and ZFS partitions by hand:

# Load the ZFS kernel module:
`kldload zfs`

# Force 4K sectors:
`sysctl vfs.zfs.min_auto_ashift=12`

# Create a swap partition (in this case, of 4 GB size):
`gpart add -a 4k -l swap0 -s 4G -t freebsd-swap ada0`

# Create a ZFS partition to fill the remaining free space:
`gpart add -a 4k -l zfs0 -t freebsd-zfs ada0`

# Mount tmpfs in /mnt, where the installer will install the operating system to:
`mount -t tmpfs tmpfs /mnt`

# Create your zroot pool:
`zpool create -f -o altroot=/mnt -O compress=lz4 -O atime=off -m none zroot /dev/gpt/zfs0`

# Create ZFS file system hierarchy:
`zfs create -o mountpoint=none zroot/ROOT`
`zfs create -o mountpoint=/ zroot/ROOT/default`

# Define the default ZFS datasets for root zpool:
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp # We'll be using tmpfs for /tmp.
`zfs create -o mountpoint=/usr -o canmount=off zroot/usr`
`zfs create zroot/usr/home`
`zfs create -o setuid=off zroot/usr/ports`
`zfs create zroot/usr/src`
`zfs create -o mountpoint=/var -o canmount=off zroot/var`
`zfs create -o exec=off -o setuid=off zroot/var/audit`
`zfs create -o exec=off -o setuid=off zroot/var/crash`
`zfs create -o exec=off -o setuid=off zroot/var/log`
`zfs create -o atime=on zroot/var/mail`
`zfs create -o setuid=off zroot/var/tmp`

# Create a symlink to /usr/home:
`ln -s /usr/home /mnt/home`

# Change /tmp and /var/tmp permissions:
chmod 1777 /mnt/tmp # Not needed now.
`chmod 1777 /mnt/var/tmp`

# Configure the Boot Environment:
`zpool set bootfs=zroot/ROOT/default zroot`

# Instruct FreeBSD to mount ZFS pools during system initialization:
`printf 'zfs_enable="YES"' >> /tmp/bsdinstall_etc/rc.conf`

# Add an fstab entry to use a GELI-encrypted swap partition:
`printf "/dev/gpt/swap0.eli\tnone\tswap\tsw\t0\t0\n" >> /tmp/bsdinstall_etc/fstab`

# Add an fstab entry to use tmpfs for /tmp:
`printf "tmpfs\t/tmp\ttmpfs\trw,mode=1777\t0\t0\n" >> /tmp/bsdinstall_etc/fstab`

# Mount the existing EFI partition of your disk:
`mount -t msdosfs /dev/ada0p1 /media`

# Create FreeBSD boot directory in the EFI partition:
`mkdir -p /media/efi/freebsd`

# Copy FreeBSD loader to the previously created directory:
`cp /boot/loader.efi /media/efi/freebsd/loader.efi`

# Now add a UEFI boot entry:
`efibootmgr --create --activate --label "FreeBSD" --loader "/media/efi/freebsd/loader.efi"`

# Unmount EFI partition:
`umount /media`

# Exit Shell Partitioning mode, so bsdinstall can continue and complete the installation:
`exit`


After finishing the installation, don't forget to add a boot entry for FreeBSD in GRUB or rEFInd.
Here I share some sample entries to guide you a bit. Remember to modify them to reflect your own reality:

# For GRUB:

```
menuentry "FreeBSD" {
    insmod part_gpt
    insmod fat
    set root=(hd0,gpt1)
    chainloader /efi/freebsd/loader.efi
}
```

# For rEFInd, add something like this at the end of refind.conf:

```
# FreeBSD
menuentry "FreeBSD" {
    loader /efi/freebsd/loader.efi
    icon /efi/refind/icons/os_freebsd.png
}
```

That's all. Hope you find it useful!

*Sources:*
https://www.freebsd.org/doc/handbook/bsdinstall-partitioning.html
https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
http://kev009.com/wp/2016/07/freebsd-uefi-root-on-zfs-and-windows-dual-boot/
https://github.com/freebsd/freebsd-src/blob/releng/13.0/usr.sbin/bsdinstall/scripts/zfsboot
https://github.com/freebsd/freebsd-src/blob/releng/13.0/usr.sbin/bsdinstall/scripts/bootconfig


*EDIT 1:*
-Added some sources.

*EDIT 2:*
-Reworded the title for better clarity.
-Added a note about FreeBSD's efi loader name used in this tutorial.
-Removed /tmp from ZFS datasets, to use a tmpfs /tmp instead. This way it provides better performance and doesn't cause troubles with some programs, like Firefox on Wayland, for instance.

*UPDATE (2021/02/12):*
-Fixed some little issues.
-Added useful efibootmgr command.
-Updated FreeBSD efi boot loader name and location to match new FreeBSD 13.0 defaults.
-Added more sources.


----------



## unwillexist (Oct 10, 2020)

Just followed this guide but when i issue

mount -t zfs zroot/ROOT/default /mnt

it only says device is busy. What can i do ?


----------



## patovm04 (Oct 11, 2020)

unwillexist said:


> Just followed this guide but when i issue
> 
> mount -t zfs zroot/ROOT/default /mnt
> 
> it only says device is busy. What can i do ?


Hmm what if you issue this command: `df -h`
Could you share the output?


----------



## unwillexist (Oct 11, 2020)

patovm04 said:


> Hmm what if you issue this command: `df -h`
> Could you share the output?











						33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01
					

Image 33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01 hosted in ImgBB




					ibb.co


----------



## unwillexist (Oct 11, 2020)

unwillexist said:


> 33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01
> 
> 
> Image 33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01 hosted in ImgBB
> ...



I do not use swap btw. That is only change i made on your guide.


----------



## patovm04 (Oct 11, 2020)

unwillexist said:


> 33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01
> 
> 
> Image 33-BCB3-A6-A427-4-E35-82-B9-3-E697-B483-C01 hosted in ImgBB
> ...


Well, not sure about that "Device busy" message, but your output shows your zroot/ROOT/default is already mounted in /mnt , and that's all that really matters. So you can go on with the next steps now


----------



## amnixed (Nov 28, 2020)

patovm04 said:


> Hello patovm04,


I ran into a couple of problems following your guide:


patovm04 said:


> # Instruct FreeBSD to mount ZFS pools during system initialization:
> `echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf`


`# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory`

`mkdir /mnt/etc` or create zfs filesystem for /etc?



patovm04 said:


> # Mount the existing EFI partition of your disk:
> `mount -t msdosfs /dev/ada0p1 /media`


gpart(8) says ada0p1 is the type ms-recovery (NTFS?) and mount(8) fails. Does the Windows 10 installer even have the option for FAT (=msdosfs?)?


----------



## twllnbrck (Nov 28, 2020)

patovm04 said:


> Essentially, you will be installing FreeBSD with root-on-ZFS on the remaining free space of the disk,


Which advantage do you have to install FreeBSD with root on zfs when you dual boot from the same disk? Im just wondering ..


----------



## patovm04 (Nov 28, 2020)

twllnbrck said:


> Which advantage do you have to install FreeBSD with root on zfs when you dual boot from the same disk? Im just wondering ..


To be able to take advantage of ZFS most useful features such as boot environments, as far as I understand.


----------



## patovm04 (Nov 28, 2020)

amnixed said:


> I ran into a couple of problems following your guide:
> 
> `# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
> bash: /mnt/etc/rc.conf: No such file or directory`
> ...


Hmm it's difficult to help you if you're not following the steps to the letter (for instance, you are running bash which isn't even part of FreeBSD's live install media).
Also it's uncommon EFI not to be your first partition, but given that's your case, it's just a matter of figuring out the correct partition number (/dev/ada0p3 for example).


----------



## amnixed (Nov 28, 2020)

patovm04 said:


> Hmm it's difficult to help you if you're not following the steps to the letter (for instance, you are running bash which isn't even part of FreeBSD's live install media).
> Also it's uncommon EFI not to be your first partition, but given that's your case, it's just a matter of figuring out the correct partition number (/dev/ada0p3 for example).


You are right, I was probably inattentive following your guide (which is actually excellent), gave up too soon and proceeded with an MBR/UFS installation. Will try it again!


----------



## 313 (Dec 11, 2020)

Thanks to the OP for the excellent tutorial. Following the instructions I was able to get dual boot going without any problems. I did however notice a couple of things that need correction, should one follow the steps by the letter.

1) The following is conflicting:



> # Mount a tmpfs to /mnt, where the installer will install the operating system into:
> `mount -t tmpfs tmpfs /mnt`
> ...
> # Mount your pool:
> `mount -t zfs zroot/ROOT/default /mnt`



If tmpfs is mounted on /mnt, the latter mount command causes the 'device busy' error that some users noticed. The tmpfs mount command should be skipped.

2) This does not work as it is, because /mnt/etc does not exist:



> # Instruct FreeBSD to mount ZFS pools during system initialization:
> `echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf`



Instead, I did
`echo 'zfs_enable="YES"' >> /tmp/bsdinstall_etc/rc.conf`
..and the results were as expected.


----------



## patovm04 (Dec 11, 2020)

313 said:


> Thanks to the OP for the excellent tutorial. Following the instructions I was able to get dual boot going without any problems. I did however notice a couple of things that need correction, should one follow the steps by the letter.
> 
> 1) The following is conflicting:
> 
> ...


Hmm interesting , as I did follow it to the letter before posting it here and didn't have those issues. Maybe something's changed since then... If I recall correctly I used FreeBSD 12.1-RELEASE at the time.
Well, I guess the how-to needs some updating now. Gonna do it some time in the future after checking it thoroughly. Thank you


----------



## Zamana (Jan 1, 2021)

Hi!

Can you tell me if your guide would work in the case of 2 separated HDDs?

On my laptop, I already have Windows 10 installed in the first HDD (for work), and I would like to install FreeBSD in the second HDD (for personal use).

Thanks.


----------



## drewlander (Jan 4, 2021)

Thank you for the writeup! Well done!

Just for anyone else who runs into this:
I also ran into:

```
# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
bash: /mnt/etc/rc.conf: No such file or directory
```
making the directory then re-running worked for me

After copying boot1.efi, I made a directory in EFI specifically for FreeBSD then ran:

```
efibootmgr -c -l /mnt/EFI/FREEBSD/BOOTX64.EFI -L FREEBSD
```

working like a charm


----------



## patovm04 (Feb 12, 2021)

drewlander said:


> Thank you for the writeup! Well done!
> 
> Just for anyone else who runs into this:
> I also ran into:
> ...


Thanks, I just updated the guide. Those issues should be solved now


----------



## patovm04 (Feb 12, 2021)

Zamana said:


> Hi!
> 
> Can you tell me if your guide would work in the case of 2 separated HDDs?
> 
> ...


Not sure, I've never done it to be honest...


----------



## scottro (Feb 19, 2021)

I just did this on FreeBSD-13.0-BETA2. I use CentOS-8-stream grub to boot it. (Though I had to lock grub at an earlier version as later versions weren't booting FreeBSD, though I had a different custom.cfg, as that was on a UFS partition).
Anyway, this time, got an error that it couldn't find the file, hit any key to continue, which I did, and it booted FreeBSD properly. So, anyway, thanks again, I think this is kind of neat to have a ZFS partition dual booting with Linux. It does, aside from giving people the chance to use bectl or beadm, also give them a chance to experience ZFS stability and such, though in laptop use, I don't know if it will be noticeable.


----------



## scottro (Mar 7, 2021)

For the sake of accuracy, I eventually figured this out. I'd put insmod part gpt (a space, rather than an underline) so the error was that it couldn't find part.mod. As I mentioned if I hit any key to continue, it still worked and if I eliminated the insmod  line for part_gpt it worked without giving an error. However, once I figured out what I'd done wrong, I corrected the part<space>gpt to read part_gpt and then all was good. Thanks again for this tutorial.


----------



## scottro (Mar 9, 2021)

Our of curiosity, I tried this with 14-CURRENT. (One nice thing about CURRENT on my hi res yoga2, is that it doesn't seem to need the drm-kmod--even during installation the fonts are nice and readable. 

However, with CURRENT the method doesn't work. Instead, when I choose shell at the partitioning section, I get an error that it's unable to create, The exact error is

```
Could not create directory /mnt/user/freebsd-dist
```
Then goes on to say an installation step has failed. 
This is a CURRENT snapshot from 2021-03-14. I realize that mentioning CURRENT is more or less taboo, but as I'd be installing it in lieu of Linux, it is also an I like FreeBSD instead of Linux thread, and they can go on for pages.  

 I'm wondering if it can be worked around by dropping to shell right at installation, but my knowledge of ZFS is far less than yours, and I'm not confident of what commands I'd run.  And of course, as it's CURRENT, the next snapshot might fix it.  One other nice thing about having it on a zfs partition, I realized, is that if something freezes, which sometimes happens on this laptop, and I have to manually power it off, ZFS would be, I think, less likely to have data corrupted. (Though there's nothing important on this particular laptop). \


----------



## Phishfry (Mar 9, 2021)

scottro said:


> I'm wondering if it can be worked around by dropping to shell right at installation


There is little that can't be done at the post install shell.
You can even download packages into the new install.
But the pre-install shell has drawbacks. No disk is yet selected and no internet available.
It is good for starting modules like Chelsio cards that need a loader.conf setting for starting up.
The shell during 'disk partition' phase is good for concocting a graid1 array. I could see the same for custom ZFS.


----------



## scottro (Mar 10, 2021)

Thanks.  I also tried setting up a ZFS partition pre-install, but didn't know what I was doing well enough to get it working that way. As it's a CURRENT snapshot, hopefully, I"ll have better luck with the next one.


----------



## Phishfry (Mar 10, 2021)

One thing to mention with the bsdinstall 'partition' phase shell.
Notably I had to make some command line settings by hand for my gmirror arrangement.



```
echo "/dev/mirror/gm0p2 / ufs rw 1 1" >> /tmp/bsdinstall_etc/fstab
echo 'geom_mirror_load="YES"' >> /tmp/bsdinstall_boot/loader.conf
```


----------



## scottro (Mar 19, 2021)

To update, trying again on the latest CURRENT, using a snapshot from 2021-03-18 (yesterday), your method once again works perfectly for me.
I'll also make a comment, that some did ask the purpose of doing this, and your answer is certainly good, that one can take advantage of beadm or bectl. I have also found, on this particular laptop, sometimes the system has frozen, if say, I made a typo in .xinit and sometimes, I have had to shut it by holding down the power switch.  While nothing has ever been hurt by that, having something in ZFS makes me less nervous about shutting the laptop down that way, though even if the whole drive was zfs, I wouldn't recommend the method.


----------



## Kalero (Mar 30, 2021)

Is it possible to make it work on a BIOS/MBR environment?


----------



## SirDice (Mar 30, 2021)

As far as I know it's not possible to directly boot ZFS with MBR, there's simply not enough room in the master boot record. Note that a BIOS boot doesn't mean you're stuck with the MBR partitioning scheme. Many older machines will boot just fine from a GPT partitioned disk.


----------



## Kalero (Mar 31, 2021)

SirDice said:


> As far as I know it's not possible to directly boot ZFS with MBR, there's simply not enough room in the master boot record. Note that a BIOS boot doesn't mean you're stuck with the MBR partitioning scheme. Many older machines will boot just fine from a GPT partitioned disk.


Exactly, that's what I guessed. And if I'm not wrong, as far as there's a Windows installation already present on the disk (with a MBR partition scheme), I think I can not change the partition scheme.


----------



## marcinkk (Mar 31, 2021)

Kalero said:


> Is it possible to make it work on a BIOS/MBR environment?





Kalero said:


> Exactly, that's what I guessed. And if I'm not wrong, as far as there's a Windows installation already present on the disk (with a MBR partition scheme), I think I can not change the partition scheme.



Hi

I had the similar problem today.
Windows installed on MBR disk.
Free space made.
I wated to install FreeBSD.
On ZFS if possible.

I read:
https://wiki.freebsd.org/RootOnZFS#Using_MBR_Disk
And then:
Installing FreeBSD Root on ZFS using UFS /boot
Installing FreeBSD Root on ZFS using FreeBSD-ZFS partition in a FreeBSD MBR Slice
and the first post from this thread.

It took me some time to understand what I am doing, a few failed tries, but I've done it finally 

I've decided to boot from UFS and run from ZFS then.

A little howto below (explanation in the linked instructions and in the first post):

Start installer, choose the "Shell" option in "Partitiong", and then:

_create partitions_

in my case:

`# gpart add -s 1G -t freebsd-ufs da0s3
# gpart add -s 16G -t freebsd-swap da0s3
# gpart add -t freebsd-zfs da0s3`

and the result is:

`% gpart show da0
=>       32  488212768  da0  MBR  (233G)
         32       2016       - free -  (1.0M)
       2048     204800    1  ntfs  [active]  (100M)
     206848  125622272    2  ntfs  (60G)
  125829120  362383680    3  freebsd  (173G)

% gpart show da0s3
=>        0  362383680  da0s3  BSD  (173G)
          0    2097152      1  freebsd-ufs  (1.0G)
    2097152   33554432      2  freebsd-swap  (16G)
   35651584  326732096      4  freebsd-zfs  (156G)`

_add bootcode_

`# gpart bootcode -b /boot/boot ad0s3`

_load required ZFS modules_

`# kldload opensolaris.ko
# kldload zfs.ko`

_now, like in the first post_



> Create your zroot pool:
> `# zpool create -f -o altroot=/mnt -O compress=lz4 -O atime=off -m none zroot /dev/da0s3d`
> 
> Create ZFS file system hierarchy:
> ...



_add partition for boot data_

`# mkdir /mnt/bootdir
# newfs /dev/da0s3a
# mount /dev/da0s3a /mnt/bootdir
# mkdir /mnt/bootdir/boot
# cd /mnt
# ln -s bootdir/boot /mnt/boot
# chflags -h sunlink /mnt/boot`

_modify configuration files_

`# echo 'zfs_enable="YES"' > /tmp/bsdinstall_etc/rc.conf
# echo 'zfs_load="YES"' > /tmp/bsdinstall_boot/loader.conf
# echo 'vfs.root.mountfrom="zfs:zroot/ROOT/default"' >> /tmp/bsdinstall_boot/loader.conf`

and add to /tmp/bsdinstall_etc/fstab


```
# Device                       Mountpoint              FStype  Options         Dump    Pass#
/dev/ad0s3a                    /bootdir                ufs     rw              0       0
/dev/ad0s3b                    none                    swap    sw              0       0
```

_Type  exit and continue system install._

After restart I made:

`# gpart set -a active -i 1 da0`

Next restart, Windows Server 2008 R2 started, and I added FreeBSD to Windows Boot Manager:

Dual Booting – Booting FreeBSD using Windows Boot Manager

I hope I didn't forget anything...

Regards,
Marcin


----------

