# HOWTO: Install FreeBSD 9.0 RELEASE (Root on UFS + ZFS, RAID1)



## vand777 (Jan 21, 2012)

I'll show how to install FreeBSD 9.0 RELEASE on the machine with 2 disks (RAID 1). I'll use the following logical layout:


```
UFS 1G   /           ro
ZFS *    /home       rw
ZFS *    /tmp        rw
UFS *    /usr        ro | softupdates (mounted r/w only for packages updates)
ZFS *    /usr/obj    rw 
ZFS *    /usr/ports  rw 
ZFS *    /usr/src    rw 
ZFS *    /var        rw
UFS 1G   /var/db/pkg ro | softupdates (mounted r/w only for packages updates)
```


----------



## vand777 (Jan 21, 2012)

*(1) Boot from a FreeBSD9 installation DVD or memstick and choose "Live CD".*

*(2) Load the necessary kernel modules.*

```
# kldload geom_mbr.ko
# kldload geom_mirror.ko
# kldload opensolaris.ko
# kldload zfs.ko
```

*(3) Create the necessary partitions on the disks.*

```
# sh
# disk0=da0
# disk1=da1
# gpart create -s gpt $disk0
# gpart add -b 34 -s 94 -t freebsd-boot $disk0
# gpart add -s 1G -t freebsd-ufs $disk0
# gpart add -s 4G -t freebsd-swap $disk0
# gpart add -s 1G -t freebsd-ufs $disk0
# gpart add -s 8G -t freebsd-ufs $disk0
# gpart add -t freebsd-zfs $disk0
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 $disk0
# gpart backup $disk0 | gpart restore -F $disk1
```

*(4) Create RAID1 for UFS and swap filesystems.*

```
# gmirror label  root  ${disk0}p2 ${disk1}p2
# gmirror label  swap  ${disk0}p3 ${disk1}p3
# gmirror label  pkg   ${disk0}p4 ${disk1}p4
# gmirror label  usr   ${disk0}p5 ${disk1}p5
```


----------



## vand777 (Jan 21, 2012)

*(5) Create filesystems and mount them.*

```
# newfs -m 1 /dev/mirror/root
# newfs -m 1 -U /dev/mirror/pkg
# newfs -m 1 -U /dev/mirror/usr
# mkdir /tmp/NEWROOT
# mount /dev/mirror/root /tmp/NEWROOT
# mkdir /tmp/NEWROOT/usr
# mount /dev/mirror/usr /tmp/NEWROOT/usr
# zpool create -m /tmp/NEWROOT/basefs basefs mirror ${disk0}p6 ${disk1}p6
# zfs create -o mountpoint=/tmp/NEWROOT/home basefs/home
# zfs create -o mountpoint=/tmp/NEWROOT/var basefs/var
# zfs create -o mountpoint=/tmp/NEWROOT/tmp basefs/tmp
# zfs create -o mountpoint=none basefs/usr
# zfs create -o mountpoint=/tmp/NEWROOT/usr/obj basefs/usr/obj
# zfs create -o mountpoint=/tmp/NEWROOT/usr/ports basefs/usr/ports
# zfs create -o mountpoint=/tmp/NEWROOT/usr/src basefs/usr/src
```

P.S. Feel free to play with ZFS filesystems options and create additional filesystems if you need/want.

*(6) Install FreeBSD.*

```
# cd /usr/freebsd-dist
# export DESTDIR=/tmp/NEWROOT
# for file in base.txz lib32.txz kernel.txz;
# do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
```


----------



## vand777 (Jan 21, 2012)

*(7) Provide basic configuration.*

```
# cat > /tmp/NEWROOT/etc/fstab << __EOF__             
#dev                #mount        #fs  #opts #dump #pass
/dev/mirror/root    /             ufs  rw    1     1     
/dev/mirror/swap    none          swap sw    0     0  
/dev/mirror/usr     /usr          ufs  rw    2     2  
basefs/var          /var          zfs  rw    0     0
/dev/mirror/pkg     /var/db/pkg   ufs  rw    2     2  

basefs              /basefs       zfs  rw    0     0
basefs/home         /home         zfs  rw    0     0
basefs/usr/obj      /usr/obj      zfs  rw    0     0
basefs/usr/ports    /usr/ports    zfs  rw    0     0
basefs/usr/src      /usr/src      zfs  rw    0     0
basefs/tmp          /tmp          zfs  rw    0     0
__EOF__   

# cat > /tmp/NEWROOT/boot/loader.conf << __EOF__      
zfs_load="YES"                                  
ahci_load="YES"                                 
geom_mirror_load="YES"                          
__EOF__  

# cat > /tmp/NEWROOT/etc/rc.conf << __EOF__
zfs_enable="YES"
__EOF__
```

*(8) Unmount filesystems and reboot.*

```
# cd /
# zfs umount -a
# umount /tmp/NEWROOT/usr
# umount /tmp/NEWROOT
# zfs set mountpoint=/home basefs/home
# zfs set mountpoint=/var basefs/var
# zfs set mountpoint=/tmp basefs/tmp
# zfs set mountpoint=/usr/obj basefs/usr/obj
# zfs set mountpoint=/usr/ports basefs/usr/ports
# zfs set mountpoint=/usr/src basefs/usr/src
# zpool export basefs
# reboot
```


----------



## vand777 (Jan 21, 2012)

*(9) Boot FreeBSD in single user mode.*

```
# mount -w /
# zpool import -f basefs
# reboot
```

*(10) Set root password*

```
# passwd
```

*(11) Set hostname*

```
# echo hostname=\"HOSTNAME\" >> /etc/rc.conf
```

*(12) Set timezone*

```
# tzsetup
```

*(13) Move termcap into /etc (instead of useless link on crash)*

```
# rm /etc/termcap
# mv /usr/share/misc/termcap /etc
# ln -s /etc/termcap /usr/share/misc/termcap
```

*(14) Setup networking and do any other necessary changes to /etc.*

*(15) Add latest security patches.*

```
# freebsd-update fetch
# freebsd-update install
```

*(16) Set / to be mounted read-only in /etc/fstab.*

```
[color="Green"]+/dev/mirror/root /      ufs  ro    1     1   [/color]
[color="Red"]-/dev/mirror/root /      ufs  rw    1     1[/color]
```

*(17) Reboot and enjoy.*

```
# reboot
```


----------



## vand777 (Jan 21, 2012)

Reserved 1


----------



## vand777 (Jan 21, 2012)

Reserved 2


----------



## vand777 (Jan 21, 2012)

Reserved 3


----------



## vand777 (Jan 21, 2012)

Reserved 4


----------



## phoenix (Jan 21, 2012)

Just a quick note: if you have a separate /home, then you don't need the symlink.


----------



## vand777 (Jan 21, 2012)

phoenix said:
			
		

> Just a quick note: if you have a separate /home, then you don't need the symlink.



Thank you. Will remove it from the guide in a second.


----------



## RusDyr (Jan 24, 2012)

I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later. 
This zfs partitioning way is in almost all howtos. Does anybody think the sense of it?


----------



## _martin (Jan 24, 2012)

RusDyr said:
			
		

> I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later.
> This zfs partitioning way is in the almost each howtos. Does anybody think the sense of it?



With ZFS, these tasks are much easier indeed; using UFS-only setup you have to do it beforehand. But I think it's _the_ perfect place to do it during installation. You're not on time trial when installing new BSD, are you? It only takes couple of seconds to do it right.


----------



## RusDyr (Jan 25, 2012)

What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.


----------



## gkontos (Jan 25, 2012)

RusDyr said:
			
		

> What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.



There is a difference because if you want to set specific properties to those datasets, compression for example, you have to do it before they get populated. 

/usr/src  is a good candidate for compression but if you set this after the installation then you pretty much don't have any.

/usr/ports is another good candidate. But /usr/ports/distfiles is not since it usually contains compressed files.

Of course all those are suggestions and anybody can improvise.

Regards


----------



## _martin (Jan 25, 2012)

RusDyr said:
			
		

> What's the point? There is no difference in /usr/ports, /usr/obj and so on at least in current ZFS configuration.



For example, when you do a /usr snapshot. I don't care about /usr/ports/* , /usr/src/* at all. 
You don't need /usr/ports for kernel debugging; /usr/obj can be shared to others too (when possible). So it does make sense to separate it.


Reasons for separate FS? 

* backup/restore (do I need to back it up?) - restore time
* mount options (performance and/or security) 
* implications when FS gets full or corrupted (can it be restored online ?)


----------



## bbzz (Jan 25, 2012)

All above and zfs list show quickly how much space is taken in each fs


----------



## RusDyr (Jan 26, 2012)

Yeah, I thought about snapshots after I've wrote it. 
Freeing space by compression or deduplication isn't valuable, especially on systems where are ZFS is (storage servers, isn't it?).


----------



## gkontos (Jan 26, 2012)

RusDyr said:
			
		

> Yeah, I thought about snapshots after I've wrote it.
> Freeing space by compression or deduplication isn't valuable, especially on systems where are ZFS is (storage servers, isn't it?).



Your original question was why all guides create so many ZFS datasets during installation, instead of doing this afterwards.

This has nothing to do with snapshots whatsoever. The only reason why you might want to create a dataset before installing the OS, is because you want to assign certain properties such as, compression, dedup, copies, etc. 

I hope this clarifies your question.


----------



## beesatmsu (Jan 26, 2012)

I am trying this now. First time I tried the 
	
	



```
# kldload geom_mbr.ko
```
 It gave me a bunch of errors. however after I tried the second line, then went back to try it again, it says file aleady exists. So I hope it was loaded. 

Now I am stuck at 
	
	



```
# gpart backup $disk0 | gpart restore -F $disk1
```
 because it says 
	
	



```
gpart: geom 'ada1': Operation not permitted
```

dmesg says I have ada0 and ada1, both 1TB drives.  It appears ada1 is not writable.


----------



## beesatmsu (Jan 26, 2012)

So I proceeded as if I had a single disk and ignored disk1. I had to omit 'mirror' and ${disk1} in the *zpool* command.

But now I tried exactly:


```
# cd /usr/freebsd-dist
# export DESTDIR=/tmp/NEWROOT
# for file in base.txz lib32.txz kernel.txz;
# do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
```

and got:


```
cat: lib32.txz: no such file or directory.
```

I am using a freebsd FreeBSD 9.0 bootdisk (i386).


----------



## beesatmsu (Jan 26, 2012)

indeed, ls gives:

```
MANIFEST
doc.txz
kernel.txz
src.txz
base.txz
games.txz
ports.txz
```

but no lib32.txz.


----------



## _martin (Jan 26, 2012)

gkontos said:
			
		

> The only reason why you might want to create a dataset before installing the OS, is because you want to assign certain properties such as, compression, dedup, copies, etc.



Now that's quite some harsh statement (especially "the only" part).


----------



## gkontos (Jan 27, 2012)

matoatlantis said:
			
		

> Now that's quite some harsh statement (especially "the only" part).



Ok, would like to elaborate?


----------



## beesatmsu (Jan 27, 2012)

Oops, just found out my boot disk was 9.0-RC3, not 9.0-release as I thought. Will try again tomorrow.


----------



## kpa (Jan 27, 2012)

You won't find lib32.txz on i386 install disk, it's amd64 only.


----------



## _martin (Jan 27, 2012)

gkontos said:
			
		

> Ok, would like to elaborate?



I think I did in my post above. You are right though - several other ZFS properties can be set, but it's far from being *the only* reason. And that's why it caught my eye.


----------



## beesatmsu (Jan 27, 2012)

Thanks.  Should I use amd64? Most of my computers are quadcore amd CPUs. But I used i386 just in case. I have one Intel CPU but it is used for windows. 

What is the advantages for using amd64? i386 still works for amd CPU fine.

Zach



			
				kpa said:
			
		

> You won't find lib32.txz on i386 install disk, it's amd64 only.


----------



## estrabd (Jan 27, 2012)

This is very nice, thank you so much!

Question - and this might be a little dense - but is there a write-up anywhere that describes use cases for ZFS and regular maintenance, etc? I am about to be setting up a new server, and would very much like to accurately determine if ZFS/RAID1 is the right set up for me.  I'm just a little fuzzy on *what* that gives me exactly. Thanks!


----------



## beesatmsu (Jan 27, 2012)

I am using the amd64 9.0 release today. I still get the same error as yesterday after `kldload geom_mbr.ko`

```
g_dev_taste: make_dev_p() failed (gp->name=ada0s1, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1a, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1b, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1d, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1e, error=17)
g_dev_taste: make_dev_p() failed (gp->name=ada0s1f, error=17)
```


----------



## beesatmsu (Jan 27, 2012)

Strange, I cannot *gpart ada0* (operation not permitted), nor can I *gpart ada1* (File exists). If I try to zero ada1, it says operation not permitted!

I have ada0 (ad6, SATA, 1TB) and ada1 (ad8, SATA, 1 TB). 

On an AMD Phenom 9650 CPU with 4 GB memory. Booted with FreeBSD AMD64 9.0 Release (DVD) in Live CD mode.


----------



## vand777 (Jan 27, 2012)

What is the result of *gpart show*? If it already shows something, then you can delete the existing settings with the following commands:

```
# gpart destroy -F ada0
# gpart destroy -F ada1
```

And then start from the beginning.

P.S. Sorry, was in a business trip for the whole week.


----------



## vand777 (Jan 27, 2012)

RusDyr said:
			
		

> I never can understand why do you need create such a numbers of [zfs] partitions like /usr/ports, /usr/obj/, /usr/ports/distfiles and so on. It's excessive, doesn't neccesary for quick installation and obviusly can be done later.
> This zfs partitioning way is in almost all howtos. Does anybody think the sense of it?



In my guide /usr is sitting on UFS filesystem. If I need /usr/ports to be located on ZFS partition, then I have to do these steps. Yes, they can be done later... But why to postpone it to a later stage?


----------



## beesatmsu (Jan 27, 2012)

vand777, thanks.

*gpart* shows my partitions I made yesterday following your instructions but with 9.0-RC3. I get 
	
	



```
gpart: Device busy
```
 when I tried to do it for ada0. I am using a different machine but the same two HDDs now. Thought there might be something wrong with my MB. So far it seems similar.  I burned a i386 9.0_release also and will try that also today.


----------



## beesatmsu (Jan 27, 2012)

Strangely the first line does not give me error now for this machine.


----------



## beesatmsu (Jan 27, 2012)

vand777, thanks. Now I destroyed both ada0 and ada1 successfully. Last time ada1 was not connected properly (power cable messup).  But the geom_mbr error came back. It does not seem to affect things though. I am trying to create partitions now.


----------



## beesatmsu (Jan 27, 2012)

When I tried to do 
	
	



```
gpart backup $disk0 | gpart restore -F $disk1
```

I got:


```
gpart: geom 'ada1': Operation not permitted
```

This seems to be the same error as yesterday (different boot disk, different machine, but the same two drives). It is possible the two drives are different brands (but same cylinders etc) and this caused the error? Freebsd FreeBSD 8.1 did not have a problem with gmirror with two different drives.


----------



## beesatmsu (Jan 27, 2012)

I am giving up on AMD64. Got another error with 
	
	



```
gmirror label root ${disk0}p2
```
 I omitted disk1 since it won't allow me anything. The error I got was 
	
	



```
gmirror: Can't store metadata on ada0p2: Operation not permitted
```

Strangely, *gpart show* says ada1 now is gone (I destroyed it once), but now *gpart* wont find it.


----------



## beesatmsu (Jan 27, 2012)

I started from scratch using the i386 disk (DVD). Still got the same error when I tried to 
	
	



```
gmirror label root ${disk0}p2 ${disk1}p2
```

I got: 
	
	



```
gmirror: Can't store metadata on ada0p2: Operaton not permitted.
```

I am about to give up zfs on 9.0.


----------



## beesatmsu (Jan 27, 2012)

The errors (which happened from p2 to p5) did not seem to affect other things, but now after finishing my network does not work. Yesterday the i386 9.0 RC3 worked fine with network, rc.conf/hosts/resolv.conf files under /etc all looked fine.


----------



## beesatmsu (Jan 27, 2012)

It appears the MB's ethernet card is gone while I was installing it. Put another PCI card in, not working. Another one, still not working.  Having bad luck today.


----------



## beesatmsu (Jan 30, 2012)

I figured out (finally!) that freebsd FreeBSD did not like my wireless router (the wired outlets worked fine with Win, but not with freebsd FreeBSD) -- I figured out because last time a working disk did not have network either. Today I moved the machine to a netgear switch and now network is working...(but Freebsd FreeBSD 9.0 did not see fxp0 but 8.0 saw it).

How do I turn on gmirror? Right now gmirror shows only the 8 gig partition as the master. 

*gmirror list*


```
Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: ada1p5
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 1770742616
```
gpart show

```
=>        34  1953525101  ada0  GPT  (931G)
          34          94     1  freebsd-boot  (47k)
         128     2097152     2  freebsd-ufs  (1.0G)
     2097280     8388608     3  freebsd-swap  (4.0G)
    10485888     2097152     4  freebsd-ufs  (1.0G)
    12583040    16777216     5  freebsd-ufs  (8.0G)
    29360256  1924164879     6  freebsd-zfs  (917G)

=>        34  1953525101  ada1  GPT  (931G)
          34          94     1  freebsd-boot  (47k)
         128     2097152     2  freebsd-ufs  (1.0G)
     2097280     8388608     3  freebsd-swap  (4.0G)
    10485888     2097152     4  freebsd-ufs  (1.0G)
    12583040    16777216     5  freebsd-ufs  (8.0G)
    29360256  1924164879     6  freebsd-zfs  (917G)
```


----------



## beesatmsu (Jan 30, 2012)

I also got these messages in dmesg:


```
GEOM_MIRROR: Device mirror/root launched (1/1).
GEOM_MIRROR: Device mirror/swap launched (1/1).
GEOM_MIRROR: Device mirror/pkg launched (1/1).
GEOM_MIRROR: Device mirror/usr launched (1/1).
GEOM_MIRROR: Device root already configured.
GEOM_MIRROR: Device swap already configured.
GEOM_MIRROR: Device pkg already configured.
GEOM_MIRROR: Device mirror/urs launched (1/1).
GEOM_MIRROR: Device root already configured.
GEOM_MIRROR: Device swap already configured.
GEOM_MIRROR: Device pkg already configured.
```


----------

