# How to Install FreeBSD 8.1 w/Mirrored System and ZFS RAID Home



## Ruler2112 (Jan 17, 2011)

This will detail how to set up FreeBSD 8.1 (amd64) over 3 hard drives.  The system itself is on a partition that's mirrored (using gmirror) twice so any of the drives can be used to boot the system.  Swap is distributed across all three drives for efficiency.  The majority of the space is used for a raidz (RAID 5) ZFS file system.  The use of various (though certainly not all) options available in ZFS are shown.

DISCLAIMER: I'm using the settings and drives that exist on the box I just got done with.  Yours may be different - adjust the specifics as needed.  I obtained some of this information from the internet, but mostly by experimenting on my own.  It works perfectly for me, but may not for you.  I'm not responsible for any data loss, fights, wars, apocalyptic disasters, spontaneous combustion of either yourself or others, defenestration, brain liquification, invasion of paranormal mind-controlling levitating eyeball-creatures from another dimension, or any other damages that may arise from the use or misuse of this information.  In other words, you are responsible for your own actions.


```
1.  Wipe the start of all three drives with binary 0s to start clean.
    1a.  Boot from Linux LiveCD
    1b.  [file]dd if=/dev/zero of=/dev/sda bs=1M count=100[/file]
    1c.  [file]dd if=/dev/zero of=/dev/sdb bs=1M count=100[/file]
    1d.  [file]dd if=/dev/zero of=/dev/sdc bs=1M count=100[/file]

2.  Start the installation.
    2a.  Boot from the FreeBSD 8.1 amd64 install DVD.
    2b.  Choose your country.
    2c.  Begin the Standard install process.

3.  Create partitions.
    3a.  Select the first drive, ad8, from the menu.
    3b.  Create a 25000m partition, type 165.
    3c.  Create a 3869m partition, type 165.
    3d.  Create a partition for the rest of the space, type 165.
    3b.  Hit q to exit the partition manager.
    3e.  Select BootMgr to install a boot manager.
    3f.  Repeat steps 3b through 3e for the second and third drives shown, ad10 and ad12.
         (Later it'll be important that the partitions themselves exist.)

4.  Create slices in the partitions.
    4a.  Create a single slice in ad8s2, ad10s2, and ad12s2; designate them as swap.
    4b.  Create a 3000m slice in ad8s1; filesystem is /
    4c.  Create a 2000m slice in ad8s1; filesystem is /tmp
    4d.  Create a 10000m slice in ad8s1; filesystem is /usr
    4e.  Create another slice in ad8s1; filesystem is /var
         (Don't bother creating slices inside ad10 or ad12 aside from swap in s2).

5.  Install system from the DVD.  I usually install everything but the ports, which I download the latest version of later.

6.  Go through the configuration, entering what you want for your system.

7.  Reboot, again booting from the install DVD.
    (This is needed because the file systems on ad8 are mounted after the install.)
    (For whatever reason, they cannot be umounted from the shell after installation.)
    (A reboot gets around this.)

8.  Enter a shell.
    8a.  Select FixIt from the main menu.
    8b.  Choose the option to use the file system from the DVD.

9.  Create the mirror.
    9a.  [file]ln -s /dist/boot/kernel /boot[/file]
         (This is to allow gmirror to work when running from the DVD.)
         (If you try to load gmirror without this, you'll see that it won't work.)
    9b.  [file]gmirror load[/file]
    9c.  [file]gmirror label -v gm0s1 /dev/ad8s1[/file]

10. Set system to use mirror at startup.
    10a. [file]mount /dev/mirror/gm0s1a /mnt[/file]
    10b. [file]echo geom_mirror_load="YES" >> /mnt/boot/loader.conf[/file]
    10c. Edit /mnt/etc/fstab and change each of the 4 file systems (/, /tmp, /usr, and /var) from /dev/ad8s1x to /dev/mirror/gm0s1x

11. Reboot.  Boot into the Linux LiveCD again.

12. Remove the bootable flag from all but the first partition.
    12a. [file]fdisk /dev/sda[/file]
    12b. a[ENTER]2[ENTER]a[ENTER]3[ENTER]w[ENTER]
         (This toggles the activate flag on partition 2 and 3, then writes out the changes.)
    12c. Repeat 12a and 12b for /dev/sdb and /dev/sdc.

13. Reboot again, this time boot from the hard drive.

14. Make the mirror have more than one disk.
    14a. [file]gmirror insert gm0s1 /dev/ad10s1[/file]
    14b. [file]gmirror insert gm0s1 /dev/ad12s1[/file]
    14c. [file]gmirror status[/file]
    14d. Wait for a while, repeating 14c every few minutes.
    14e. When the status shows complete, the system is now mirrored across all three drives.

15. Enable ZFS at startup.
    15a. [file]echo zfs_enable="YES" >> /etc/rc.conf[/file]

16. Create the ZFS pool.
    16a. [file]zpool create tank raidz ad8s3 ad10s3 ad12s3[/file]

17. Create mount points inside of ZFS.  Each mount point may have different properties.
    17a. [file]zfs create tank/home[/file]
    17b. [file]zfs create tank/home/ruler[/file]
    17c. [file]zfs create tank/home/ruler/programs[/file]
    17d. [file]zfs create tank/home/virtualprinter[/file]

18. Set various options on the different ZFS mount points.
    18a. [file]zfs set compression=gzip tank/home/virtualprinter[/file]
         (/home/virtualprinter is made up of almost exclusively text files.)
         (Because of this, compression saves considerable space.)
    18b. [file]zfs set copies=2 tank/home/ruler/programs[/file]
         (The system will keep 2 copies of programs I've written.)
         (This is in addition to the RAID redundancy.)
    18c. [file]zfs set atime=off tank[/file]
         (I really don't care when the files were last accessed.)
         (Turning off the recording of such information improves performance.)
         (This one line affects the all the mount points in the entire ZFS.)

19. Get ZFS home ready.
    19a. [file]cp -p /home/ruler/.* /tank/home/ruler[/file]
         (The -p is important to preserve the file permissions.)
    19b. [file]chown -R ruler:users /tank/home/ruler[/file]

20. Begin using new ZFS home directory.
    20a. [file]rm /home[/file]
         (/home is a symbolic link to /usr/home on FreeBSD 8.1.)
         (You can look in /usr/home/ruler for the original files afterwards if you want.)
         (Because it is a symbolic link, rm -r or rmdir is not needed.)
    20b. [file]zfs set mountpoint=/home tank/home[/file]
    20c. Enjoy!
```


I know I should have used cmd tags instead of file, but cmd automatically does a line break around the tag and screws up the indentation.  I put the whole thing in a code block for the same reason - spaces don't show up correctly otherwise.  Apologies in advance to DD.  

I hope this helps somebody.


----------



## chainsaw (Jan 25, 2011)

I am trying to set up my system using this howto and can't get past the 10th point, when I type 
[cmd=]mount /dev/ad4s1a /mnt[/cmd]
I get 
	
	



```
no such file or directory
```
 however I made a slice ad4s1a witch is / partition. 

Maybe I made a mistake at point 4.b when I created all slices in ad4s1 because it says just create in ad4, without saying in witch partition of ad4 (in my case) and how big should the /var partition be? I used the entire left space (10000MB).


----------



## Ruler2112 (Jan 26, 2011)

You're right on with step 4 missing the s1 at the end of the dev node.  I've fixed the original post - thanks for the heads-up on it.  I intentionally left the size of /var blank because it does consume the rest of the drive.  Entering a specific size would not only be redundant, but you'd have to play with the numbers until you got it just right because drive geometries differ slightly.  It's about 10 gig, which is exactly what you have.

I don't know why it'd come up with that error.  I'd think that either the device node or the directory doesn't exist.  What do you get if you do an [cmd=]ls -l /dev/ad4*[/cmd] and [cmd=]ls -l /[/cmd]?  If /mnt doesn't exist, that's easy enough to fix.  If the dev node doesn't exist, I don't know what to say...


----------



## eyescream (Feb 10, 2011)

Just wanted to thank you for this howto!

It answered some questions I had and gave me a nice idea of how to set up my FreeBSD system. Much appeciated.


----------



## gettons (Feb 27, 2011)

great howto.
I think I ll try it out.

One question. Why did you use gmirror instead of zfs for the OS ? is that not possible ?


----------



## Ruler2112 (Feb 28, 2011)

gettons said:
			
		

> great howto.
> I think I ll try it out.
> 
> One question. Why did you use gmirror instead of zfs for the OS ? is that not possible ?



Thank you.  

From my research, you can use ZFS as the root file system and boot from it, but support for this is quite new and you can boot from it only if it's not any type of RAID.  Since I know gmirror is solid, I used that.  Originally, my plan was to have the first partition of the third disk be a static backup, but wondered what would happen if I inserted another partition into the mirror and sure enough, I had a 3 disk mirror.


----------



## Ruler2112 (Sep 29, 2011)

Installed a new system this past week and discovered a change from the above.  Updated Step 10a from:


```
10a. [file]mount /dev/ad8s1a /mnt[/file]
```

to


```
10a. [file]mount /dev/mirror/gm0s1a /mnt[/file]
```

for FreeBSD 8.2.


----------

