# Howto FreeBSD 8.1 + geli + zfs + large disks



## hend0012 (Mar 16, 2011)

*HOWTO FREEBSD 8.1 + GELI + ZFS + LARGE DISKS*

I encountered this problem for the first time, when I had to install a server with a Raid 5 6Tb config. Every time I installed FreeBSD it just created slices of 1.5 Tb, I wanted to create a small FreeBSD installation and create a big slice (5,5 Tb) for backup purposes. So I went on a search on the web. I wasn't lucky at all, this can mean two things, or I am doing something very wrong or there wasn't a solution yet. So I got creative and came up with the solution you see below. If you have a better way of doing this please let me know.

*My setup consists of:*

mfid0 5,5 Tb â€“ Will be split in 2 GB Swap 1 GB / and the remaining will be for ZFS.
I will be using a GPT table because of the large disks.

mfid0p3 / 1G
mfid0p2 swap 2G
mfid0p4 everything else
*What are we going to need:*

You will have to make a FreeBSD usb installation disk. You can download the usb img here. 


```
dd if=DIR/FreeBSD-8.1-RELEASE-amd64-memstick.img of=/dev/sdb bs=64k
```

And you will need a FreeBSD installation Livefs disc. You can download the installation Livefs disc here.

*Installation and setup*

The steps:
1. Boot Livefs disk and drop to Fixit shell.
2. Setup partitions/slices on the internal disks.
3. Install FreeBSD.
4. Boot into OS modify partitions copy OS reboot into new OS.
5. Setup GELI on the disk.
6. Create the ZFS pool.
7. Mount ZFS in the right place.
8. Copy OS to ZFS tank.
9. Reboot and done.

*Step 1 : Boot Livefs disk and drop to Fixit shell.*

First plug in the FreeBSD usb and boot in to the Livefs cd and start the Fixit, (Where using a Livefs disk because not all servers boot the usb disk versions correctly).

*Step 2 : Setup partitions/slices on the internal disks.*

Create a new Partition table on mfid0 we are using GPT 

```
# gpart create -s GPT mfid0
```
Create FreeBSD boot partition to contain the boot code. 

```
# gpart add -t freebsd-boot -s 128 mfid0
```
Write protective MBR to sector 0 of disk 

```
# gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptboot -i 1 mfid0
```
Create swap 

```
# gpart -t freebsd-swap -s 2G mfid0
```
Create  / 

```
# gpart -t freebsd-ufs -s 1G mfid0
```
Create  /everything 

```
# gpart -t freebsd-ufs mfid0
```
Show partitions


```
# gpart show

              34  11717836733  mfid0  GPT  (5.5T)
      34          128      1  freebsd-boot  (64K)
    162      8388608      2  freebsd-swap  (2.0G)
 8388770      2097152      3  freebsd-ufs  (1.0G)
10485922  11707350845      5  freebsd-ufs  (5.5T)
```

*Step 3 : Install FreeBSD 8.1.*

Format ufs 

```
# newfs mfid0p3
# newfs mfid0p4
```

Mount Partition

```
# mount /dev/mfid0p3 /mnt
```

Install FreeBSD 

```
# cd /mnt
# mkdir usbstick
# mount /dev/ad0a(FreeBSD usb) /mnt/usbstick
# cd /mnt/usbstick/8.1-RELEASE/base
# ./install.sh
# cd /mnt/usbstick/8.1-RELEASE/kernels
# ./install.sh GENERIC
```

Copy GENERIC kernel 

```
# cd /mnt/boot
# rmdir kernel
# cp -Rp GENERIC kernel
```

Create new fstab 

```
# vi /mnt/etc/fstab

/dev/mfid0p3       /       ufs        rw          1        1

# reboot
```

*Step 4 : Boot into OS modify partitions copy OS reboot into new OS.*

After the reboot we are going to remove swap

```
# gpart delete -i 2 mfid0
```
Now we are going to create a 2Gb ufs partition 

```
# gpart add -t freebsd-ufs -s 4G mfid0
```
This will be mfid0p5 
Mount mfid0p5 

```
# mount /dev/mfid0p5 /mnt
```
Tar Filesystem 

```
# cd /
```


```
# tar cf - --one-file-system * | tar xpf - -C /mnt
```


```
# vi /mnt/etc/fstab
/dev/mfid0p5       /       ufs        rw          1        1
```
Now reboot and when starting the boot loader press space because we want to boot from different partition 

```
# 0:ad(0,p5)/boot/loader
```

*Step 5 : Setup GELI on the disks.*

When you are booted in to the new installation we can start making the crypt and zfs tank

Create crypt

```
# geli init -b -v -s 4096 /dev/mfid0p4
```
Attach disk

```
# geli attach /dev/mfid0p4
```

*Step 6 : Create the ZFS pool.*

Create zfs tank

```
# zpool create tank mfid0p4.eli
```

*Step 7 : Mount ZFS in the right place.*

Format mfid0p3

```
# newfs /dev/mfid0p3
```
Mount mfid0p3

```
# mkdir /tank/bootdir
# mount /dev/mfid0p3 /tank/bootdir
```

*Step 8 : Copy OS to ZFS tank.
*
Copy boot

```
# cp -Rp /boot /tank/bootdir/
```
Create symlink

```
# ln -sf /tank/boot /tank/bootdir/boot
```
Edit loader

```
# vi /tank/boot/loader.conf

zfs_load="YES"
geom_eli_load="YES"
```
Edit fstab

```
# vi /tank/bootdir/etc/fstab

tank      /    zfs     rw     0     0
```
We have a working /boot now, so let's fill up the ZFS root

```
# cd /

# tar cf - --one-file-system * | tar xpf - -C /tank
```
We are now going to restore the swap partition that we earlier removed to use as ufs

```
# gpart delete -i 5 mfid0

# gpart add -t freebsd-swap -s 4G mfid0
```
We need to update fstab

```
# vi /tank/etc/fstab

tank            /           zfs     rw     0     0

/dev/mfid0p3     /bootdir    ufs     rw     1     1

/dev/mfid0p2    swap        swap    sw     0     0
```
Final step is to prevent ZFS from re-mounting tank, as it will be used to root filesystem. This can be done by

```
# zfs set mountpoint=legacy tank
```

*Step 9 : Reboot* 


```
# reboot
```


----------



## Miklos (Mar 31, 2011)

Excellent guide, but it would be better to export the disks as single drives and let ZFS handle the raid.

Hardware raid and ZFS overlaps and does not make sense (unless you raid several hardware raids).


----------

