# Makefile for Vermaden's FreeBSD ZFS root install adapted for



## J65nko (Aug 8, 2013)

Introduction

What does the Makefile do? The comments at the beginning of the file gives an explanation:


```
# This BSD makefile merely automates Vermaden's "'ZFS madness / boot
# environments" mirrored disk setup.
# 
# For a complete description of this particular ZFS only setup see below.
# 
# The ZFS configuration described by Vermaden does not align the
# disk partitions for 'Avanced Format' 4K disks and also does not
# instruct ZFS to use 4K blocks for the write and read operations.
#
# The procedure implemented here does align the EFI/gpt partitions
# used on a 4K (8 x 512) # sector boundary. By  using the 'gnop' trick
# the FreeBSD ZFS implementation is coached  to use an 'ashift' value
# of 12 (2^12 = 4096)
# This value makes ZFS read and write on 4K boundaries
# 
# Credits for the ZFS install procedure:
#
# Slawomir Wojciech Wojtczak (vermaden)
#   [url]http://forums.freebsd.org/showthread.php?t=31662[/url] 
#   [url]http://www.daemonforums.org/showthread.php?t=7099 [/url]
#
# Sources and /credits for the 4K alignment issue and gnop hack:
# 
# Warren Block (wblock@) for his numerous posts about partition alignment: 
#     [url]http://forums.freebsd.org[/url] (Installation and Storage sections)
#     [url]http://www.wonkity.com/~wblock/docs/html/disksetup.html#_the_new_standard_gpt[/url]
# Ivan Voras for the gnop workaround:
#     [url]http://ivoras.sharanet.org/blog/tree/2011-01-01.freebsd-on-4k-sector-drives.html[/url] 
# George Kontostanos (gkontos) for using gnop and a ZFS cache file: 
#    [url]http://forums.freebsd.org/showthread.php?t=23544[/url]
#    [url]http://www.aisecure.net/2012/01/16/rootzfs/[/url]
```

The advantage of using a makefile scripted approach:

No more typing or pasting errors. Especially if you wipe and partition the wrong disk because of a typo.
Speed. After having set the configurable settings you can do a reproducible FreeBSD ZFS install within a few minutes.
Easy testing of your changes by using memory disks.
If one of your makefile modifications halts with an error you can fix it and redo it easily.
There are utilities to wipe all partitioning info from the disks, to destroy the ZFS pool and ZFS datasets and to remove the memory disks. Because they only operate on the devices defined in the makefile, you are less prone to make mistakes.

Overview of the Makefile targets 

The utilities:


*show*
Display all Makefile variables
Check presence/existence of installation file sets
Check availability of a template for rc.conf

*diskinfo* -  run diskinfo(8) on the disks 
*zfsload* - load ZFS kernel modules 
*pool_destroy* - destroy the ZFS datasets and pool
*gpart_destroy* - remove existing partitioning from disks
*md_create* - create swap backed memory disks
*md_destroy* - remove the memory disks

The pre-installation tasks can be performed individually or by using the *pre_install* target:


*partition*
destroy existing partitioning and initializes a fresh GPT partition scheme
create *freebsd-boot* and *freebsd-zfs* partitions on a 4K boundary and labels them
write appropriate bootcode to disk

Creating a ZFS pool with ashift=12 (2^12 = 4096):

*gnop4k* - create gnop devices with 4k sectors for the disks
*pool4k* - create a ZFS storage pool with the gnop devices
*export* - export the storage pool
*gnop_destroy* - destroy the gnop devices
*import* - import the storage pool with the original disks
*chk_ashift* - run `# zdb` to verify an ashift value of 12 

*zfs_options* - set several ZFS options
*zfs_fs* - create the ZFS data sets
*zfs_swap* - configure ZFS swap space

The actual installation:

*install* - extract the installation file sets base.txz and kernel.txz

The post-installation tasks done by target *post_install*: 

*loader.conf* - initialize the loader.conf file
*fstab* - create an empty /etc/fstab file
*rc.conf*
copy rc.conf template  
add ZFS magic to rc.conf 

*zfs_boot* - copy the cachefile to /zfs/boot
*zfs_umount* - unmount the ZFS datasets
*mountpoint* - set the ZFS mountpoint to 'legacy' so that ZFS will not handle the mounts automagically. This way these can be done manually or with Vermaden's port of the Solaris sysutils/beadm utility.

For the really lazy ones among us there is a target 'all' that does it all:

```
pre_install: trailer partition gnop4k pool4k export gnop_destroy import \
        chk_ashift zfs_options zfs_fs zfs_swap

install: trailer
        for THIS in ${INSTALL_SETS} ; do \
        tar --unlink -xvpJf $${THIS} -C ${MOUNT} ;\
        done
        ls -l ${MOUNT}
        zpool status ${POOL}
        zpool iostat ${POOL}
        zfs list

post_install: trailer loader.conf fstab rc.conf zfs_boot zfs_umount mountpoint

[color=blue]all[/color]: trailer pre_install install post_install
```

Configuration

By changing the makefile variables you can adapt the installation to your wishes. Things like the pool name, the disks used etc. 


```
POOL            = super
ROOT_SET        = ${POOL}/ROOT
BOOT_SET        = ${ROOT_SET}/default

# --- installation file sets
SETS            = base.txz kernel.txz
DIR             = /usr/freebsd-dist     # On FreeBSD liveCD/DVD/memstick
.for X in ${SETS}
INSTALL_SETS += ${DIR}/${X}
.endfor
```

If you specify memory or RAM disks:

```
DISKTYPE        = ada
DISKTYPE        = md

.if ${DISKTYPE} == "md"

# --- memory disks see md(4) and mdconfig(8)
# size of disk needs to be at least 64M else you encounter this error:
# "cannot create 'pool': one or more devices is less than the minimum size (64M)"

MD_SIZE         = 2g
SWAPSIZE        = 256m
DISK_1          = /dev/md1
DISK_2          = /dev/md2
DISKS           = ${DISK_1} ${DISK_2}
# --- gpt/EFI labels
LABEL_ZFS       = mdisk_
LABEL_BOOT      = mdboot
```
For non-memory disks:

```
.else

# --- real spinning rust disks
DISK_1          = /dev/ada1
DISK_2          = /dev/ada2
DISKS           = ${DISK_1} ${DISK_2}
SWAPSIZE        = 4G
# --- gpt/EFI labels
LABEL_ZFS       = SYSTEM_
LABEL_BOOT      = BOOT_

.endif
```

On the FreeBSD live CD the /tmp and /mnt directories are writeable.


```
TMP             = /tmp
CACHEFILE       = ${TMP}/zpool.cache
TEMPLATE_RC_CONF= ${TMP}/template_rc.conf
MOUNT           = /mnt
RC_CONF         = ${MOUNT}/etc/rc.conf
LOADER_CONF     = ${MOUNT}/boot/loader.conf
FSTAB           = ${MOUNT}/etc/fstab
```

Check these settings with:


```
[cmd=#]make show[/cmd]
-------------------------
current variable settings
-------------------------
DEBUG                : []
POOL                 : [xpool]
ROOT_SET             : [xpool/ROOT]
BOOT_SET             : [xpool/ROOT/default]
SETS                 : [base.txz kernel.txz]
DIR                  : [/usr/freebsd-dist]
INSTALL_SETS         : [/usr/freebsd-dist/base.txz /usr/freebsd-dist/kernel.txz]
DISKTYPE             : [ada]
MD_SIZE              : []
SWAPSIZE             : [4G]
DISK_1               : [/dev/ada1]
DISK_2               : [/dev/ada2]
DISKS                : [/dev/ada1 /dev/ada2]
LABEL_ZFS            : [disk_]
LABEL_BOOT           : [boot_]
SWAPSIZE             : [4G]
LABEL_ZFS            : [disk_]
LABEL_BOOT           : [boot_]
TMP                  : [/tmp]
CACHEFILE            : [/tmp/zpool.cache]
MOUNT                : [/mnt]
TEMPLATE_RC_CONF     : [/tmp/template_rc.conf]
RC_CONF              : [/mnt/etc/rc.conf]
LOADER_CONF          : [/mnt/boot/loader.conf]
FSTAB                : [/mnt/etc/fstab]
-------------------------
```
After displaying these values there will be check for the installation files:


```
Checking for installation sets ...
Set /usr/freebsd-dist/base.txz : found!
Set /usr/freebsd-dist/kernel.txz : found!
```

The last check is for a rc.conf template file. 

```
Checking for 'rc.conf' template ...
/tmp/template_rc.conf not found
exit 100
*** [show] Error code 100
```
If the file is found we get this:

```
----------- rc.conf template ----------
cat /tmp/template_rc.conf
# ------ rc.conf
#defaultrouter=192.168.222.10
hostname='althusser.utp.xnet'
network_interfaces='bge0'
ifconfig_bge0="SYNCDHCP"

# --- daemons
inetd_enable=NO
sshd_enable=yes
sendmail_enable=yes

# --- time synchronization
ntpd_enable=YES
#openntpd_enable=YES
```

Make sure that these essential files are present before you continue with the installation.


----------



## J65nko (Aug 8, 2013)

Pre-installation steps

For demonstration purpose two memory or RAM disks will be used.


```
[cmd=#]make md_create[/cmd]

Creating Memory disk devices /dev/md1 /dev/md2:
Memory disk devices:
md1     swap     2048M
md2     swap     2048M
crw-r-----  1 root  operator    0, 162 Aug  5 16:50 /dev/md1
crw-r-----  1 root  operator    0, 163 Aug  5 16:50 /dev/md2
crw-------  1 root  wheel       0,  63 Aug  5 14:58 /dev/mdctl
-------------end of md_create --------------------
```

Running `# diskinfo` on these disks.


```
[cmd=#]make diskinfo[/cmd]
if [ -e /dev/md1 ] ; then diskinfo -v /dev/md1 ; fi
/dev/md1
        512             # sectorsize
        2147483648      # mediasize in bytes (2.0G)
        4194304         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset

if [ -e /dev/md2 ] ; then diskinfo -v /dev/md2 ; fi
/dev/md2
        512             # sectorsize
        2147483648      # mediasize in bytes (2.0G)
        4194304         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset

-------------end of diskinfo --------------------
```

It is not always needed to edit the Makefile to behave differently.
To use this target on one of my disks reporting a 512 sectorsize to the OS but having  a stripesize of 4096, I can override the makefile variable *DISKS* on the command line:

```
[cmd=#]make DISKS=/dev/ada1 diskinfo[/cmd]

if [ -e /dev/ada1 ] ; then diskinfo -v /dev/ada1 ; fi
/dev/ada1
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        [color=blue]4096[/color]            # stripesize
        0               # stripeoffset
        3876021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        S1E160MR        # Disk ident.

-------------end of diskinfo --------------------
```

After this intermezzo we start the real pre-installation work by creating 4K aligned GPT partitions.  

*WARNING*: the following command will remove all partitioning information from the disks as defined in the makefile. Please make sure that this is correct and double-check the settings you configured by running `# make show`. You will lose data if the settings refer to the wrong disks.


```
[cmd=#]make partition[/cmd]

if gpart show /dev/md1 ; then  gpart destroy -F /dev/md1 ; fi
gpart: No such geom: /dev/md1.
gpart create -s gpt /dev/md1
md1 created
NR=$( echo /dev/md1 | tr -c -d '0-9' ) ; gpart add  -b 40 -s 128k -t freebsd-boot -l mdboot${NR} /dev/md1 ; gpart add              -t freebsd-zfs  -l mdisk_${NR}  /dev/md1
md1p1 added
md1p2 added
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 /dev/md1
bootcode written to md1
gpart show /dev/md1
=>     34  4194237  md1  GPT  (2.0G)
       34        6       - free -  (3.0k)
       40      256    1  freebsd-boot  (128k)
      296  4193975    2  freebsd-zfs  (2G)
[snip]
ls -l /dev/gpt
crw-r-----  1 root  operator    0, 170 Aug  5 16:56 mdboot1
crw-r-----  1 root  operator    0, 176 Aug  5 16:56 mdboot2
crw-r-----  1 root  operator    0, 168 Aug  5 16:56 mdisk_1
crw-r-----  1 root  operator    0, 174 Aug  5 16:56 mdisk_2
-------------end of partition --------------------
```

Here we see that the partitions start at sector 40 and 296. A quick command line modulo calculation with bc(1) confirms that the start sectors on a 8 x 512 = 4096 boundary. An alternative method, a division shows no remainder.


```
[cmd=$]echo '296 % 8' | bc[/cmd]
0
[cmd=$]echo 'scale=4 ; 296 / 8' | bc[/cmd]  
37.0000
```

Creating the gnop(8) devices with 4K sectors.


```
[cmd=#]make gnop4k[/cmd]

Creating gnop devices with 4K sectors .....
for X in /dev/md1 /dev/md2 ; do  NR=$( echo ${X} | tr -c -d '0-9' ) ;
 gnop create -S 4096 /dev/gpt/mdisk_${NR} ; done

ls -l /dev/gpt
crw-r-----  1 root  operator    0, 170 Aug  5 16:56 mdboot1
crw-r-----  1 root  operator    0, 176 Aug  5 16:56 mdboot2
crw-r-----  1 root  operator    0, 168 Aug  5 16:56 mdisk_1
crw-r-----  1 root  operator    0, 166 Aug  5 17:38 mdisk_1.nop
crw-r-----  1 root  operator    0, 174 Aug  5 16:56 mdisk_2
crw-r-----  1 root  operator    0, 172 Aug  5 17:38 mdisk_2.nop

gnop list
Geom name: gpt/mdisk_1.nop
WroteBytes: 0
ReadBytes: 131072
Writes: 0
Reads: 21
Error: 5
WriteFailProb: 0
ReadFailProb: 0
Offset: 0
Providers:
1. Name: gpt/mdisk_1.nop
   Mediasize: 2147311616 (2G)
   Sectorsize: [color=blue]4096[/color]
   Mode: r0w0e0
Consumers:
1. Name: gpt/mdisk_1
   Mediasize: 2147315200 (2G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 151552
   Mode: r0w0e0

[snip]

gnop status
           Name  Status  Components
gpt/mdisk_1.nop     N/A  gpt/mdisk_1
gpt/mdisk_2.nop     N/A  gpt/mdisk_2
-------------end of gnop4k --------------------
```

Note the gnop GEOM provider with a sector size of 4096.
A ZFS storage pool with these freshly created 4K devices is the next step.


```
[cmd=#]make pool4k[/cmd]

Creating zpool with 4K gnop devices
if [ -f /tmp/zpool.cache ] ; then mv /tmp/zpool.cache /tmp/zpool.cache.prev ; fi
ls -l /dev/gpt/*nop
crw-r-----  1 root  operator    0, 166 Aug  5 17:38 mdisk_1.nop
crw-r-----  1 root  operator    0, 172 Aug  5 17:38 mdisk_2.nop

# ---- creating a ZFS mirror pool  without automatically mounting it (-m none) .....
zpool create -f -o cachefile=/tmp/zpool.cache -m none  super mirror /dev/gpt/mdisk_*nop

zpool list super
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
super  1.98G   560K  1.98G     0%  1.00x  ONLINE  -

zpool status super
  pool: super
 state: ONLINE
  scan: none requested
config:

        NAME                 STATE     READ WRITE CKSUM
        super                ONLINE       0     0     0
          mirror-0           ONLINE       0     0     0
            gpt/mdisk_1.nop  ONLINE       0     0     0
            gpt/mdisk_2.nop  ONLINE       0     0     0

errors: No known data errors

zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
super   468K  1.95G   144K  none
-------------end of pool4k --------------------
```

Export the pool:


```
[cmd=#]make export[/cmd]
Exporting super ......
---------------------------------
zpool export super
---------------------------------
Showing import status of super ......
---------------------------------
zpool import
   pool: super
     id: 4218734048000312785
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        super                ONLINE
          mirror-0           ONLINE
            gpt/mdisk_1.nop  ONLINE
            gpt/mdisk_2.nop  ONLINE
-------------end of export --------------------
```

Destroy the gnop devices.


```
[cmd=#]make gnop_destroy[/cmd]
Destroy the 4K gnop devices
for X in /dev/md1 /dev/md2 ; 
  do  NR=$( echo ${X} | tr -c -d '0-9' ) ;
   gnop destroy /dev/gpt/mdisk_${NR}.nop ;
done

ls -l /dev/gpt
total 0
crw-r-----  1 root  operator    0, 170 Aug  5 16:56 mdboot1
crw-r-----  1 root  operator    0, 176 Aug  5 16:56 mdboot2
crw-r-----  1 root  operator    0, 168 Aug  5 16:56 mdisk_1
crw-r-----  1 root  operator    0, 174 Aug  5 16:56 mdisk_2
[snip]
-------------end of gnop_destroy --------------------
```

After the gnop devices have gone the way of the dodo, we now import our pool.


```
[cmd=#]make import[/cmd]

Import the pool
zpool import -o cachefile=/tmp/zpool.cache super
zpool list super
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
super  1.98G   720K  1.98G     0%  1.00x  ONLINE  -
---------------------------------
zpool status super
  pool: super
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        super       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            md1p2   ONLINE       0     0     0
            md2p2   ONLINE       0     0     0

errors: No known data errors
---------------------------------
mount
/dev/ada0s3a on / (ufs, local, noatime, soft-updates)
devfs on /dev (devfs, local, multilabel)
tmpfs on /tmp (tmpfs, local)
-------------end of import --------------------
```

The pool is ONLINE but not yet mounted.

Note that for some reason the import does not use the GPT labels, but the partition numbers. I only noticed this when I test with memory disks. With real disks the import uses the labels:


```
[cmd=#]ls -l /dev/gpt[/cmd]
total 0
crw-r-----  1 root  operator    0, 173 Jul 26 04:41 boot_1
crw-r-----  1 root  operator    0, 182 Jul 26 04:41 boot_2
crw-r-----  1 root  operator    0, 171 Jul 26 04:41 disk_1
crw-r-----  1 root  operator    0, 180 Jul 26 04:41 disk_2

[cmd=#]make import[/cmd]
Import the pool
zpool import -o cachefile=/var/tmp/zpool.cache super
zpool status
  pool: super
 state: ONLINE
  scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        super           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/disk_1  ONLINE       0     0     0
            gpt/disk_2  ONLINE       0     0     0
```

Verify that the imported pool although created with 4K size gnop devices, has retained the proper *ashift* value.


```
[cmd=#] make chk_ashift[/cmd]

Verify that ashift value is 12 (2^12 = 4096 ; 2^9 = 512)
=========================================================
zdb -C -U /tmp/zpool.cache super

MOS Configuration:
        version: 28
        name: 'super'
        state: 0
        txg: 65
        pool_guid: 4218734048000312785
        hostid: 556313802
        hostname: 'althusser.utp.xnet'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 4218734048000312785
            children[0]:
                type: 'mirror'
                id: 0
                guid: 16715166418299641908
                metaslab_array: 30
                metaslab_shift: 24
                [color=blue]ashift: 12[/color]
                asize: 2142502912
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 5669787805994321979
                    path: '/dev/md1p2'
                    phys_path: '/dev/md1p2'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 10458960657003691384
                    path: '/dev/md2p2'
                    phys_path: '/dev/md2p2'
                    whole_disk: 1
                    create_txg: 4
====================================
zdb -C -U /tmp/zpool.cache super | grep ashift
                [color=blue]ashift: 12[/color]
-------------end of chk_ashift --------------------
```

That looks fine. With a 4K optimized ZFS pool, consisting of 4K aligned GPT partitions we can resume the original procedure outlined by Vermaden, by setting options that will be inherited by the child ZFS datasets.


----------



## J65nko (Aug 8, 2013)

*Pre-installation (continued)*


```
[cmd=#]make zfs_options[/cmd]
zfs set mountpoint=none super
zfs set checksum=fletcher4 super
zfs set atime=off super

zpool list super
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
super  1.98G   780K  1.98G     0%  1.00x  ONLINE  -

zpool status super
  pool: super
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        super       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            md1p2   ONLINE       0     0     0
            md2p2   ONLINE       0     0     0

errors: No known data errors
zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
super   504K  1.95G   144K  none
-------------end of zfs_options -------------------
```

For handling the boot environments with Vermaden's sysutils/beadm utility we create the following ZFS datasets and set the appropriate options:


```
[cmd=#]make zfs_fs[/cmd]
---------------------------------
zfs create                    super/ROOT
zfs create -o mountpoint=/mnt super/ROOT/default
zpool set bootfs=super/ROOT/default  super
zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super                912K  1.95G   144K  none
super/ROOT           288K  1.95G   144K  none
super/ROOT/default   144K  1.95G   144K  /mnt
---------------------------------
mount
/dev/ada0s3a on / (ufs, local, noatime, soft-updates)
devfs on /dev (devfs, local, multilabel)
tmpfs on /tmp (tmpfs, local)
[color=blue]super/ROOT/default on /mnt (zfs, local, noatime, nfsv4acls)[/color]
-------------end of zfs_fs --------------------
```
As you can see we have now a ZFS dataset mounted on /mnt. 

In the original procedure the ZFS swap space is configured after rebooting into the new system, but let us do it now.


```
[cmd=#]make zfs_swap[/cmd]

zfs create -V 256m   super/swap
zfs set org.freebsd:swap=on super/swap
zfs set checksum=off        super/swap
zfs set sync=disabled       super/swap
zfs set primarycache=none   super/swap
zfs set secondarycache=none super/swap

zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super                265M  1.69G   144K  none
super/ROOT           288K  1.69G   144K  none
super/ROOT/default   144K  1.69G   144K  /mnt
super/swap           264M  1.95G    72K  -
-------------end of zfs_swap --------------------
```

Having a ZFS pool and datasets with the super/ROOT/default mounted on /mnt we are now ready to unpack the FreeBSD installation sets.


----------



## J65nko (Aug 8, 2013)

Installation

Let us go for it.


```
[cmd=#]make install[/cmd]
for THIS in /usr/freebsd-dist/base.txz /usr/freebsd-dist/kernel.txz ;
    do  tar --unlink -xvpJf ${THIS} -C /mnt ; done
x ./
x ./bin/
x ./boot/
x ./dev/
x ./etc/
x ./lib/
x ./libexec/
x ./media/
x ./mnt/
x ./proc/
x ./rescue/
x ./root/
x ./sbin/
[snip]
x ./boot/kernel/zlib.ko
x ./boot/kernel/zlib.ko.symbols
x ./boot/kernel/linker.hints
```
After all the files have been extracted we are presented with:


directory listing of the FreeBSD installation

```
ls -l /mnt
total 146
-rw-r--r--   2 root  wheel  1014 Dec  4  2012 .cshrc
-rw-r--r--   2 root  wheel   256 Dec  4  2012 .profile
-r--r--r--   1 root  wheel  6203 Dec  4  2012 COPYRIGHT
drwxr-xr-x   2 root  wheel    46 Dec  4  2012 bin
drwxr-xr-x   7 root  wheel    39 Dec  4  2012 boot
dr-xr-xr-x   2 root  wheel     2 Dec  4  2012 dev
drwxr-xr-x  20 root  wheel    94 Dec  4  2012 etc
drwxr-xr-x   3 root  wheel    48 Dec  4  2012 lib
drwxr-xr-x   3 root  wheel     4 Dec  4  2012 libexec
drwxr-xr-x   2 root  wheel     2 Dec  4  2012 media
drwxr-xr-x   2 root  wheel     2 Dec  4  2012 mnt
dr-xr-xr-x   2 root  wheel     2 Dec  4  2012 proc
drwxr-xr-x   2 root  wheel   142 Dec  4  2012 rescue
drwxr-xr-x   2 root  wheel     6 Dec  4  2012 root
drwxr-xr-x   2 root  wheel   130 Dec  4  2012 sbin
lrwxr-xr-x   1 root  wheel    11 Dec  4  2012 sys -> usr/src/sys
drwxrwxrwt   2 root  wheel     2 Dec  4  2012 tmp
drwxr-xr-x  14 root  wheel    14 Dec  4  2012 usr
drwxr-xr-x  23 root  wheel    23 Dec  4  2012 var
```

The pool status:

```
zpool status super
  pool: super
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        super       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            md1p2   ONLINE       0     0     0
            md2p2   ONLINE       0     0     0

errors: No known data errors
```

The pool input/output statistics:

```
zpool iostat super
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
super        616M  1.38G      0      1     64  34.2K
```

listing of the ZFS data sets:

```
zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super                880M  1.09G   144K  none
super/ROOT           615M  1.09G   144K  none
super/ROOT/default   614M  1.09G   614M  /mnt
super/swap           264M  1.35G    72K  -
-------------end of install --------------------
```


Post-installation

Initializing /boot/loader.conf


```
[cmd=#]make loader.conf[/cmd]
echo 'zfs_load=YES'                          >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:super/ROOT/default"' >> /mnt/boot/loader.conf
=========== loader.conf =========================
zfs_load=YES
vfs.root.mountfrom="zfs:super/ROOT/default"
-------------end of loader.conf --------------------
```

To prevent errors at boot time an empty /etc/fstab is needed.

```
[cmd=#]make fstab[/cmd]
# ---- create empty /etc/fstab file
touch /mnt/etc/fstab
-------------end of fstab --------------------
```

Enable ZFS in /etc/rc.conf.


```
[cmd=#] make rc.conf[/cmd]
cat /tmp/template_rc.conf     >> /mnt/etc/rc.conf
cat: /tmp/template_rc.conf: No such file or directory
*** [rc.conf] Error code 1
```

We fix this error and simply redo.


```
[cmd=#]cp template_rc.conf /tmp[/cmd]
[cmd=#]make rc.conf[/cmd]
cat /tmp/template_rc.conf     >> /mnt/etc/rc.conf
echo 'zfs_enable=YES'       >> /mnt/etc/rc.conf
echo '# ------------------' >> /mnt/etc/rc.conf
=========== rc.conf =========================
# ------ rc.conf
#defaultrouter=192.168.222.10
hostname='althusser.utp.xnet'
network_interfaces='bge0'
ifconfig_bge0="SYNCDHCP"

# --- daemons
inetd_enable=NO
sshd_enable=yes
sendmail_enable=yes

# --- time synchronization
ntpd_enable=YES
#openntpd_enable=YES

zfs_enable=YES
# ------------------
```

Copy the ZFS cache file to /boot/zfs


```
[cmd=#] make zfs_boot[/cmd]
if  [ ! -d /mnt/boot/zfs ] ; then echo Cannot copy /tmp/zpool.cache to /mnt/boot/zfs ! ; exit 2 ; fi
cp -p /tmp/zpool.cache /mnt/boot/zfs/
ls -l /mnt/boot/zfs
total 1
-rw-r--r--  1 root  wheel  1488 Aug  5 17:56 zpool.cache
-------------end of zfs_boot --------------------
```

Unmount


```
[cmd=#]make zfs_umount[/cmd]
zfs unmount -a
zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super                925M  1.05G   144K  none
super/ROOT           659M  1.05G   144K  none
super/ROOT/default   659M  1.05G   659M  /mnt
super/swap           264M  1.31G    72K  -
------
mount
/dev/ada0s3a on / (ufs, local, noatime, soft-updates)
devfs on /dev (devfs, local, multilabel)
tmpfs on /tmp (tmpfs, local)
-------------end of zfs_umount --------------------
```

Set the mountpoint to *legacy*


```
[cmd=#]make mountpoint[/cmd]
ls -l /tmp
-rw-r--r--  1 root   wheel   261 Aug  5 23:35 template_rc.conf
-rw-r--r--  1 root   wheel  1488 Aug  5 17:56 zpool.cache
zfs set mountpoint=legacy super/ROOT/default
zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
super                925M  1.05G   144K  none
super/ROOT           659M  1.05G   144K  none
super/ROOT/default   659M  1.05G   659M  [color=blue]legacy[/color]
super/swap           264M  1.31G    72K  -
Setup is finished ............
----------------- done ! ---------------------------------
you can now reboot into your new ZFS only system now .....
----------------------------------------------------------
Or you could finish the elementary setup with:

1. Remount your ZFS file system    : mount -t zfs super/ROOT/default /mnt
2. Chroot into /mnt                : chroot /mnt
3. Change the root password        : passwd
4. Create the /etc/mail/aliases.db : newaliases
5. Set up your time zone           : tzsetup

6. Exit the chroot                 : exit
7. Unmount                         : umount /mnt
-------------end of mountpoint --------------------
```

That's it. A FreeBSD ZFS only system suitable for managing boot environments.


----------



## J65nko (Aug 8, 2013)

One of the first versions of the makefile without all the whistles and bells like variables and the 4K alignment procedure:


```
DISKS   = /dev/ada1 /dev/ada2
DISKS   = /dev/md1 /dev/md2

partition:
.for X in ${DISKS}
        if gpart show ${X} ; then gpart destroy -F ${X} ; fi
        gpart create -s GPT ${X}
        NUMBER=$$( echo ${X} | tr -c -d '0-9' ) ;\
        gpart add -t freebsd-boot -l bootcode$${NUMBER} -s 128k ${X} ;\
        gpart add -t freebsd-zfs -l sys$${NUMBER} ${X}
        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ${X}
.endfor

pool:
        zpool create -f -o cachefile=/tmp/zpool.cache sys mirror /dev/gpt/sys*
        zpool status
        zpool list

options:
        zfs set mountpoint=none sys
        zfs set checksum=fletcher4 sys
        zfs set atime=off sys
        zfs create sys/ROOT
        zfs create -o mountpoint=/mnt sys/ROOT/default
        zpool set bootfs=sys/ROOT/default sys

# ================================================================================
pre_install: partition pool options

install:
        cd /usr/freebsd-dist ;\
        for X in base.txz kernel.txz; do \
        tar --unlink -xpJf $${X} -C /mnt ;\
        done
        @echo Current directory: $$(pwd)

post_install: zfs_boot loader.conf fstab rc.conf umount mountpoint

all: pre_install install post_install

# ================================================================================

zfs_boot:
        cp /tmp/zpool.cache /mnt/boot/zfs/

loader.conf:
        echo 'zfs_load=YES'                              >> /mnt/boot/loader.conf
        echo 'vfs.root.mountfrom="zfs:sys/ROOT/default"' >> /mnt/boot/loader.conf

fstab:
        # ---- create empty /etc/fstab file
        touch /mnt/etc/fstab

rc.conf:
        echo 'zfs_enable=YES' >> /mnt/etc/rc.conf

umount:
        zfs list
        zfs umount -a

mountpoint:
        zfs set mountpoint=legacy sys/ROOT/default
        zfs list
```
As you can see it follows very closely the Vermaden's FreeBSD ZFS madness procedure.


----------



## J65nko (Aug 8, 2013)

*Methods to get the Makefile to the FreeBSD live CD session*

Methods to get the Makefile to the FreeBSD live CD session


Download or copy the Makefile to an USB stick and copy from the stick to the /tmp directory of the FreeBSD live CD. 

Create your own Live USB stick by doing a simple install on USB stick. Configure elementary network connectivity and ssh so you can use scp.
Then you also have the advantage of adding your favourite shell, running script(1) to produce and keep a log of your installation efforts.

Use the script from http://mfsbsd.vx.sk to create a more versatile FreeBSD live CD than the official FreeBSD installer.

Transfer the Makefile and the rc.conf template with nc(1) :
If you are on a network with DHCP you can issue an `# ifconfig` to see which network card you have: 

```
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,
                       VLAN_HWTSO,LINKSTATE>
        ether 98:4b:e1:10:08:6b
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
```

Then run `# dhclient`:

```
[cmd=#]dhclient bge0[/cmd]
DHCPREQUEST on bge0 to 255.255.255.255 port 67
DHCPACK from 192.168.222.10
bound to 192.168.222.240 -- renewal in 43200 seconds
```

Read the nc(1) manual page:


```
[b]DATA TRANSFER[/b]
     The example in the previous section can be expanded to build a basic data
     transfer model.  Any information input into one end of the connection
     will be output to the other end, and input and output can be easily cap-
     tured in order to emulate file transfer.

     Start by using [b]nc[/b] to listen on a specific port, with output captured into
     a file:

           $ nc -l 1234 > filename.out

     Using a second machine, connect to the listening [b]nc[/b] process, feeding it
     the file which is to be transferred:

           $ nc host.example.com 1234 < filename.in

     After the file has been transferred, the connection will close automati-
     cally.
```

So on the machine with the FreeBSD live CD: `# nc -l 1234 >Makefile`
On the second machine: `$ nc 192.168.222.240 1234 <Makefile`

If you have to move multiple files, it will be easier to pack  all these files into a single tar(1) archive first.


----------



## J65nko (Aug 8, 2013)

The Makefile for downloading. Please rename it to Makefile:

`# mv Makefile-ZFS-madness-1.18.txt Makefile`


----------



## J65nko (Aug 8, 2013)

*Reserved*

Reserved for further extension.


----------



## devildetail (Aug 20, 2013)

Great work, just an observation: since revision  r243502 the root pool is automatically detected by GEOM providers, hence it's not required to include the parameter vfs.root.mountfrom into /boot/loader.conf.


----------



## J65nko (Aug 20, 2013)

You also do not have to use a zpool.cache anymore, although that causes some issues with zdb(8). See the FreeBSD 9.2 zpool.cache thread.


----------



## kpa (Aug 22, 2013)

Slight correction, you still have to set the bootfs property for the pool but setting vfs.root.mountfrom is no longer needed.


----------



## Ben (Nov 18, 2013)

I copied my Makefile to a 9.2 Live-CD but I get 
	
	



```
Need an operator
```
 and 
	
	



```
Error expanding embedded variable
```
 errors. There are no unwanted line breaks or anything. Did I miss something?


----------



## J65nko (Nov 18, 2013)

The "Need an operator" error occurs if the shell commands in the Makefile are not preceded with a tab char.
From make(1):


> SHELL COMMANDS
> Each target may have associated with it a series of shell commands, nor-
> mally used to create the target.  Each of the commands in this script
> must be preceded by a tab.



Some editors replace tabs with the spaces and thus cause this error. Copying and pasting also may expands these essential tabs into spaces.


```
[cmd=#]cat Makefile[/cmd] 
ls:
        # 
        ls -ltr

# end of Makefile
[cmd=#]make[/cmd]
"Makefile", line 3: Need an operator
make: fatal errors encountered -- cannot continue

[cmd=#]cat -nt Makefile[/cmd]
     1  ls:
     2  ^I# 
     3          ls -ltr
     4
     5  # end of Makefile
```
Here you see that line 2 starts with a tab, but line 3 does not, hence the "Need an operator" error. After deleting those spaces on line 3 the Makefile runs without error.

```
[cmd=#]make[/cmd]
#
ls -ltr
total 4
drwxrwxrwt  2 root  wheel   0 Aug 23 21:53 .font-unix
drwxrwxrwt  2 root  wheel   0 Aug 23 21:53 .XIM-unix
drwxrwxrwt  2 root  wheel   0 Aug 23 21:53 .X11-unix
drwxrwxrwt  2 root  wheel   0 Aug 23 21:53 .ICE-unix
-rw-r--r--  1 root  wheel  36 Nov 19 00:18 Makefile

[cmd=#]cat -nt Makefile[/cmd]
     1  ls:
     2  ^I# 
     3  ^Ils -ltr
     4
     5  # end of Makefile
```

Instead of testing with `cat -nt` you also can use `:set list` within the vi editor. Then you will see something like this:

```
1 ls:$
      2 ^I# $
      3         ls -ltr$
      4 $
      5 # end of Makefile$
```

To make sure you did not introduce some control characters into the file you also could check with hexdump(1):

```
[cmd=#]hexdump -C Makefile[/cmd]
00000000  6c 73 3a 0a 09 23 20 0a  09 6c 73 20 2d 6c 74 72  |ls:..# ..ls -ltr|
00000010  0a 0a 23 20 65 6e 64 20  6f 66 20 4d 61 6b 65 66  |..# end of Makef|
00000020  69 6c 65 0a                                       |ile.|
```

I have no idea about the "Error expanding embedded variable" message. 

In case you cannot fix these errors send me a PM (private message) to make an arrangement for me to have a look at your file.


----------



## Ben (Nov 19, 2013)

Thanks a lot for this very (!) detailed help!

On my local computer I can see the tabs but on the server they are gone. I transferred via nc(1).


----------



## Ben (Mar 23, 2014)

*Re: Makefile for Vermaden's FreeBSD ZFS root install adapted*

I have a question about the SWAP space: The manpage says that FreeBSD does not support ZFS SWAP partitions, but as far as I can see your Makefile create the SWAP space this way, right?

Maybe I don't quite understand this SWAP limitation with ZFS.

Thanks for help.


----------



## kpa (Mar 23, 2014)

*Re: Makefile for Vermaden's FreeBSD ZFS root install adapted*

Swap on a ZFS ZVol works but may be a problem in low memory situations because there's a possibility of a race condition that quickly exhausts all remaining free memory. It can be used on a system that does not usually face such low memory situations.


----------



## Ben (Mar 23, 2014)

*Re: Makefile for Vermaden's FreeBSD ZFS root install adapted*

Ok, thanks for clarifying.


----------

