# FreeBSD 9 ZFS the easy way



## srivo (Apr 22, 2012)

Here is how I install FreeBSD 9 with ZFS. This How-To is strongly inspired by gkontos "Root On ZFS @ FreeBSD 9". I will make it a little easier by using the installer to install the base system and setup time, network and everything else.

I strongly suggest you take a look at gkontos "Root On ZFS @ FreeBSD9". He go further into optimizing zfs partition. I will keep it simple here.

Boot from a FreeBSD9 installation CD or DVD and choose *install*
Select the keyboard you like
Type the name you want for that machine
Select what you want from the distribution. You can select doc, ports, source if you like
On the partitioning screen, select *Shell*
Now we will create the ZFS partition:

```
gpart create -s gpt ada0
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
```

We will create the zfs pool mount on /mnt:

```
zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
zpool set bootfs=zroot zroot
```

Copy the zpool.cache file on the disk to be able to get it back after installation:

```
cp /var/tmp/zpool.cache /mnt/zpool.cache
```

Add swap. In that case I add 4GB of swap:

```
zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
```

Type â€œexitâ€ and go back to the let the installer finish the job
Type a password for root
Choose your network card and configure your network
Set clock and time zone
Select system configuration like sshd...
Configure dumdev if you like
Add other user if you like
Select exit to finish installation
On the manual configation screen choose *YES*, you need to copy back the zpool.cache file (very important step):

```
cp /zpool.cache /boot/zfs/zpool.cache
```

add the zfs driver:

```
echo 'zfs_enable="YES"' >> /etc/rc.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf
```

Type exit and choose *reboot*
After reboot, it's now a good idea to take your first snapshot

```
zfs snapshot zroot@config
```

You now have a working FreeBSD ZFS system and a first snapshot to rollback to in case of emergency.


----------



## srivo (May 31, 2012)

*If the system doesn't boot!*

Don't panic! You probably forgot one operation.

Boot back with the installation CD and choose "live CD" and import back your zpool:

```
zpool import -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
```
You will now have access to your zfs file system on /mnt

These is basicaly two pitfall, you forgot to copy the zpool.cache file or you forgot to load the zfs drivers.

Copy back the zpool.cache file

```
cp /var/tmp/zpool.cahe /boot/zfs/zpool.cache
```

Check back your /etc/rc.conf file to see if

```
zfs_enable="YES"
```
Is present.

Check back the /boot/loader.conf file to see if

```
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
```
are present.


----------



## TheHolm (Jun 4, 2012)

Thank you for excellent guide. Works for me. 
Is there any particular reason why you put swap on ZFS instead of bare GPT partition?


----------



## srivo (Jun 4, 2012)

No. It's just to avoid creating the /etc/fstab file.


----------



## ShelLuser (Apr 18, 2013)

*FILE tags, and it's "FreeBSD"*

First a quick thanks for posting, even after one year extremely helpful. One small addition if I may..



			
				srivo said:
			
		

> [*]We will create the zfs pool mount on /mnt:
> 
> ```
> zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
> ...


This is very good but a downside for me was that one huge ZFS filesystem can be hard to manage later; for example if you decide that you want /var on a separate filesystem. Not a problem on a separate PC, it becomes an issue if you're on a VPS or a remote server (with KVM access yet no peripheral access for example).

So, after I set up the main pool I wanted /var, /usr/ports, /usr/local, /home as well as some sub directories (/var/tmp) to be separated.

To that end I mixed directions of your tutorial with those from the main FBSD FreeBSD manual, and added my own touch (I didn't want /usr to be separate), hope this can help others as well:


```
(before the above '[I]zpool export zroot[/I]')

mkdir /mnt/usr
zfs create -o compression=lzjb -o setuid=off -o mountpoint=/usr/ports zroot/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/packages

zfs create -o mountpoint=/usr/local zroot/local

zfs create zroot/var
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o compression=on -o exec=off -o setuid=off zroot/var/mail
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
chmod 1777 /mnt/var/tmp

zfs create zroot/home
zfs create zroot/tmp
chmod 1777 /mnt/tmp
```

The twist should be obvious; by pointing the mount point for zroot/ports to /usr/ports it will still initially end up on /mnt/usr/ports because /mnt is the root for the zroot filesystem, which sits at the top of the hierarchy. After the installation is done otoh, zroot will end up on / (obviously) and thus automatically make sure that your ports filesystem also ends up on the right place (/usr/ports).

The reason I like this so far (disclaimer: with plenty of Unix experience but little specific FreeBSD experience at the time of writing) is this separates my user data (/usr/local, /var, /home) and additional or dynamic data (/usr/ports) from the system data (/, /usr, etc.).

Hope this can give others some ideas as well.


----------



## srivo (Apr 23, 2013)

I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.

But your additions are very good. PC-BSD just does them without having to type anything!


----------



## JakkFrosted (Aug 1, 2013)

Thanks for this tutorial. Only thing I had to do differently was use da0 instead of ada0. I'm running a RAID5 using a HP Smart Array card. I was able to find out the device by poking around in /dev with `fdisk`. It was my second guess at which device was actually the disk, I figured da for "disk array" :e.  I'm wondering if there's a faster way to find the disk device names?


----------



## kjs3 (Sep 1, 2013)

JakkFrosted said:
			
		

> I'm running a RAID5 using a HP Smart Array card.



You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.


----------



## ShelLuser (Sep 6, 2013)

*Be warned when using FreeBSD 9.2!*

Hi gang,

This is one of my favourite ZFS tutorials (also because I applied the steps above on all my servers) and when playing with the upcoming 9.2 version I noticed that zpool behaved differently:


```
# zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
cannot mount '/mnt/zroot': failed to create mountpoint
```
Although this looked weird to me I could also come up with a theory as to why this behaviour was changed. But taking no chances I eventually decided to write a problem report where it was confirmed that this change in behaviour is indeed intended.

So, if you like the tutorial above and are planning to use FreeBSD 9.2 (or higher) then please keep this in mind. You now need to use this:

`# zpool create -m / -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0`.

If you already used the previously mentioned zpool command and are now wondering why things don't work (and how to fix this): You don't have to start all over again, simply use this command:

`# zfs set mountpoint=/ zroot`.

This will make sure that your newly created zroot ZFS pool is now mounted under /mnt instead of /mnt/zroot.

Hope this can help some people too.

Edit: null-edit & fixed {cmd} tags gone awol afterwards.


----------



## srivo (Sep 7, 2013)

Thanks for this post!

I didn't try to install FreeBSD 9.2 this way so far!


----------



## kpa (Sep 7, 2013)

@ShelLuser. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.


----------



## ShelLuser (Sep 7, 2013)

kpa said:
			
		

> @ShelLuser. Can you check if /mnt is actually writable, the "failed to create mountpoint" error suggests that it isn't.


You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.


----------



## kpa (Sep 7, 2013)

ShelLuser said:
			
		

> You're absolutely right. In the situation above I booted with an installation CD and opted to partition the disk manually. As such the root file system is readonly.



I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.


----------



## ShelLuser (Sep 7, 2013)

kpa said:
			
		

> I remember that /tmp is writable on the installation/live CD system and can be used for creating mountpoints.


Yeah, but that's not what you want here. The reason to use /mnt is to allow the installer to extract the base system right onto your newly created ZFS structure.


----------



## JakkFrosted (Sep 8, 2013)

kjs3 said:
			
		

> You really don't want to run ZFS on top of RAID 5. ZFS expects to be talking to disks, not through a RAID 5 abstraction layer.



Well, it's not a soft RAID, though. Anyone else ever heard of this?


----------



## kpa (Sep 8, 2013)

JakkFrosted said:
			
		

> Well, it's not a soft RAID, though. Anyone else ever heard of this?



The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.


----------



## JakkFrosted (Sep 8, 2013)

kpa said:
			
		

> The point is that you don't want to have any kind of RAID, be it software or hardware, underneath ZFS. ZFS is designed around the assumption that the devices ZFS is talking to are plain disks.




Here's where I'm getting confused: Is it that having it over a RAID is _redundant_, or, is it that there are going to be undesirable issues?


----------



## J65nko (Sep 8, 2013)

See ZFS and hardware RAID


----------



## JakkFrosted (Sep 8, 2013)

J65nko said:
			
		

> See ZFS and hardware RAID



Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup _may_ preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.


----------



## kjs3 (Sep 9, 2013)

JakkFrosted said:
			
		

> Thanks. I see that the issue is data corruption VS drive failure. The RAID 5 I have set up is for drive failure, but this setup _may_ preclude ZFS from preventing data corruption -- which RAID 5 does not. Live and learn, eh? But after reading that bit of info, I feel OK with my current setup.



Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do its job. Because when you come back with "ZFS screwed up my RAID 5 array data", we're not going to be a bit sympathetic.


----------



## kjs3 (Sep 9, 2013)

srivo said:
			
		

> I don't use that way anymore. I use PC-BSD install, it's easy and does a very clean install of all the folders the right way. It also structures the folders for Beadm. You just have to choose FreeBSD install instead of PC-BSD.
> 
> But your additions are very good. PC-BSD just does them without having to type anything!



As long as you have one disk. If you want to do a ZFS install over two or more (like /var or /home on a separate disk), I can't figure a way to make the PC-BSD GUI to do it. So it's precisely no better than using this procedure.


----------



## J65nko (Sep 9, 2013)

No, you should not be OK at all in using hardware RAID 5 underneath ZFS .Some reasons:


Parity inconsistency and "RAID 5 write hole"
From RAID 5 in enterprise environments:



> As of August 2012, Dell, Hitachi, Seagate, Netapp, EMC, HDS, SUN Fishworks and IBM have current advisories against the use of RAID 5 with high capacity drives and in large arrays


----------



## wblock@ (Sep 9, 2013)

The hardware RAID card also adds another point of hardware that can fail.  And if that card fails, you may need an identical one to be able to access the RAID.


----------



## JakkFrosted (Sep 9, 2013)

kjs3 said:
			
		

> Then we can of course count on you not blame ZFS for your data corruption when your hardware RAID controller hides the disk and block level data corruption that ZFS counts on to do it's job.  Because when you come back with "ZFS screwed up my RAID-5 array data", we're not going to be a bit sympathetic.



How frequent of a problem is data corruption? I have never had to deal with it in my 7 years of server adminning. I host simple websites on my servers and a few other server programs here and there. The main reason I went with ZFS was because of the improvements on speed I read about, and, that I like to try new things. I would be willing to migrate the OS install over to a JBOD array with ZFS, but I'm not sure how to go about that. One way I'm thinking is, make a snapshot, download it, make a JBOD with ZFS, then image the array with the snapshot -- but will that snapshot work seamlessly, being that it's moving from a RAID 5 to a JBOD where ZFS is more in control. After all, you are talking as if I should heed these warnings and change things over, and I am willing to do so if I don't have to reinstall the OS from scratch and everything.



			
				wblock@ said:
			
		

> The hardware RAID card also adds another point of hardware that can fail.  And if that card fails, you may need an identical one to be able to access the RAID.



I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.


----------



## wblock@ (Sep 9, 2013)

JakkFrosted said:
			
		

> I don't see that point as valid. If I don't use the RAID card then I'm using the onboard controller, which leaves me with the same amount of hardware components.



Plain controllers do not put proprietary RAID metadata on the drives.  A RAID-Z pool can be moved to another machine with different controllers.  Arrays created with RAID controllers have metadata that may or may not work in a different machine when the original one fails.


----------



## kjs3 (Sep 10, 2013)

JakkFrosted said:
			
		

> How frequent of a problem is data corruption?



With multi-terabyte disks, corruption is inevitable.  See this report from CERN, for example.  Or this one from NEC.  Or this more mainstream one.  I could google more for you, but perhaps these will be enlightening.



			
				JakkFrosted said:
			
		

> I have never had to deal with it in my 7 years of server adminning. I  host simple websites on my servers and a few other server programs here  and there.



That you know of.  When you deal with tens of thousands of disks and petabytes of data, this sort of data corruption is patently obvious.  And that's with enterprise class drives, excellent environmentals and sophisticated monitoring.  We probably shred more failed disks a month than you've used total in seven years.  

That experience is why I use ZFS on any data I really care about.   Because I can calculate how long it's going to take statistically for it to be unrecoverable.


----------



## srivo (Sep 13, 2013)

I'm still testing the install on 9.2. The following line isn't required anymore in /boot/loader.conf for 9.2:

```
vfs.root.mountfrom="zfs:zroot"
```
zpool.cache also is not required anymore.

I don't think the following error message is a problem since we export and import back the zpool.

```
# zpool create -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0
cannot mount '/mnt/zroot': failed to create mountpoint
```
All those changes will simplify the install. I'm planning to write a new "How-to" for 9.2 once I finish testing. I will still continue testing because I have some inconsistency when I try to install with beadm and would like to have the new setup ready for it.


----------



## srivo (Sep 13, 2013)

After testing I confirm that the install works without the use of zpool.cache and the use of

```
vfs.root.mountfrom="zfs:zroot"
```
in /boot/loader.conf.

I still need to find an elegant way of creating the zpool.


----------



## kpa (Sep 13, 2013)

srivo said:
			
		

> I still need to find an elegant way of creating the zpool!



What part of the creation you're referring to here? Creating a pool is quite straightforward unless the disks are those advanced 4 KB/sector disks.


----------



## srivo (Sep 13, 2013)

In the install process when I exit to the shell to create my zpool I have trouble finding an elegant way to mount it on /mnt. I succeeded with trial and error and export/import. Sometimes it works, sometimes it fails.


----------



## kpa (Sep 13, 2013)

You can set the altroot property on creation of the pool so the mountpoints get rooted under the same directory during the installation no matter what the mountpoint properties are set for the datasets. Something like this:

`zpool create -R /mnt tank mirror ada0 ada1 ...`

This would mount the tank pool initially under /mnt/tank. You would then have to create the additional datasets and set mountpoints so that root dataset gets mountpoint of / (so it appears directly under /mnt) and delete mountpoints for the top level datasets that you don't want to be mounted.

Remember the altroot property is not saved and is reset to empty on next import or boot.


----------



## srivo (Sep 13, 2013)

The problem when I issue that command is that instead of mounting it on /mnt it mount*s* it on /mnt/tank? With an error message "can't mount..." I don't get it.


----------



## kpa (Sep 13, 2013)

That's odd. Can you post the output of the `mount` command from the shell you're using to create the pool. The mounting under /mnt/tank initially is expected because of the altroot property and mountpoint of the first created tank dataset. I should probably write a HOWTO or something but I'm not using ZFS myself at the moment.


----------



## ShelLuser (Sep 13, 2013)

Uhm guys, not to be blunt here but did you both already forgot my previous message where I explained that the zpool behaviour was changed in FreeBSD 9.2? 

I even got a PR to prove it! 
	

	
	
		
		

		
			





.


----------



## kpa (Sep 13, 2013)

Hmm I should probably set up a virtual machine with 9.2 to test this...

Edit: Yeah, I forgot. The root filesystem of the install environment is read-only and it makes it impossible to create directories under /mnt. I have to think about a way to first create the pool with altroot set to /tmp (that is on a read-write mdmfs(8)) and then switch things around so that the root dataset of the system to be installed ends up under /mnt.

Edit 2: Your PR is unfortunately invalid because it does not make a difference if zpool(8) tries to create the /tank or /mnt/tank directory on a read-only filesystem.


----------



## srivo (Sep 13, 2013)

@ShelLuser, I saw your post but since I was getting an error when I was trying to create the zpool your way, I thought you made an error in your post.

No offence!


----------



## srivo (Sep 13, 2013)

The most elegant way I found so far is this way:

```
zpool create zroot /dev/gpt/disk0
zfs set mountpoint=none zroot
zfs create zroot/ROOT
zfs create -o mountpoint=/mnt zroot/ROOT/default
zpool set bootfs=zroot/ROOT/default zroot
```
This way the system is ready for beadm. Everything works but there is still one thing I don't like. When I type `zfs list` the output tells me that the mountpoint of zroot/ROOT/default is /mnt. Of course this is normal but I didn't find that clean.


----------



## ShelLuser (Sep 13, 2013)

srivo said:
			
		

> ShelLuser, I saw your post but since I was getting an error when I was trying to create the zpool your way, I tough you made an error in your post!
> 
> No offence!


None taken what so ever, I am teasing a little bit up there (hence the smileys).

But I noticed that your example didn't seem to set the mountpoint. So, my command would be: `# zpool create -m / -R /mnt zroot /dev/gpt/disk0`. Notice the -m parameter, this sets the mount point to the root directory so that it won't try to create a new mountpoint for its own.

It could a bit confusing but -m sets the real mountpoint whereas -R sets the alternative mountpoint. zpool used to set the mountpoint automatically to root (/) but with the upcoming release it will set it automatically to /pool (where 'pool' is the name of the ZFS pool you're creating).

Hope this can help.


----------



## J65nko (Sep 13, 2013)

Because of an issue with zdb(8) I still prefer to use a cachefile. See FreeBSD 9.2 zpool.cache

For a 4K aligned pool, I do it this way:

```
# zpool history
History for 'kontos':
2013-09-11.23:07:49 zpool create -f -m none -o cachefile=/tmp/zpool.cache kontos \
                    mirror /dev/gpt/sys_1.nop /dev/gpt/sys_2.nop
2013-09-11.23:07:59 zpool export kontos
2013-09-11.23:08:46 zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache -d /dev/gpt kontos
2013-09-11.23:09:25 zpool set bootfs=kontos kontos
2013-09-11.23:09:33 zfs set checksum=fletcher4 kontos
2013-09-11.23:09:33 zfs set atime=off kontos
2013-09-11.23:09:38 zfs set mountpoint=/ kontos
2013-09-11.23:12:14 zfs create kontos/usr
2013-09-11.23:12:15 zfs create kontos/usr/home
2013-09-11.23:12:15 zfs create kontos/var
```


----------



## srivo (Sep 14, 2013)

Wait! @J65nko is right, we should avoid getting rid of zpool.cache for now. I just tried to create another boot environment with beadm and discover that beadm uses the zpool.cache file. If you have an install without the zpool.cache file just issue the following command to generate it: `zpool set cachefile=/boot/zfs/zpool.cache zroot`


----------



## srivo (Sep 14, 2013)

I still wonder if
	
	



```
zfs_enable="YES"
```
 is require*d* in /etc/rc.conf? I just installed without it and everything seem*s* to work fine.


----------



## srivo (Sep 19, 2013)

Well, the answer is: yes, we should still keep 
	
	



```
zfs_enable="YES"
```
 in our /etc/rc.conf. It starts the ZFS daemon that takes care of our ZFS drive!


----------



## kpa (Sep 19, 2013)

Not quite. There are no user space daemons for ZFS on FreeBSD. What the setting in rc.conf(5) does is to set some jail related ZFS parameters and make sure /etc/zfs/exports exists in case some of the datasets are shared with NFS using the sharenfs property.


----------



## usdmatt (Sep 19, 2013)

The rc script also does a `zfs mount -a`, which is fairly important . It mounts all ZFS file systems (those that have canmount=yes at least). Without it, no ZFS file system will be mounted unless it's your root file system, or you've added it to /etc/fstab (which you shouldn't).


----------



## fred974 (Jan 6, 2014)

Hi all,

I have followed the tutorial and when *I* typed `# zpool create -m / -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0` I get the following error:

```
mountpoint '/' exists and is not empty.
use '-m' option to provide a different default
```

Does anyone know why *I* get that message?

Thank you*.*


----------



## ShelLuser (Jan 7, 2014)

Which version of FreeBSD are you trying to install, @fred974?

Does /mnt actually exist in this particular setup?


----------



## fred974 (Jan 7, 2014)

Hi @ShelLuser,

The version I was using was FreeBSD 9.2, I didn't check if /mnt existed and I deleted the VM. I'll have to double check next time. Whilst I got your attention: in the post you mentioned that you wanted to split /usr and /var.


```
zfs create zroot/usr
zfs create zroot/usr/home
zfs create zroot/var
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs set copies=1 zroot/usr/src
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs set copies=1 zroot/usr/ports
```

Will my code above be the same as yours?

```
(before the above '[I]zpool export zroot[/I]')

    mkdir /mnt/usr
    zfs create -o compression=lzjb -o setuid=off -o mountpoint=/usr/ports zroot/ports
    zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/distfiles
    zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/packages

    zfs create -o mountpoint=/usr/local zroot/local

    zfs create zroot/var
    zfs create -o exec=off -o setuid=off zroot/var/db
    zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
    zfs create -o compression=on -o exec=off -o setuid=off zroot/var/mail
    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
    zfs create -o exec=off -o setuid=off zroot/var/run
    zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
    chmod 1777 /mnt/var/tmp

    zfs create zroot/home
    zfs create zroot/tmp
    chmod 1777 /mnt/tmp
```

Fred


----------

