# ZFS on i386 with mirrored boot and RAIDZ1 root



## volatilevoid (Jan 15, 2010)

Hi,

I'm trying to install FreeBSD with ZFS on a

- Pentium 4 3.0 GHz
- 2 GB RAM
- 3x 40 GB HDD

Unfortunately, without EFI there is no way to have GPTs, so I tried to mix the howtos from http://wiki.freebsd.org/RootOnZFS to create a mirrored zpool and a RAIDZ1 one as zfsboot is only able to boot from a mirrored zpool.

For this reason, I created one slice with three partitions (1 GB ZFS for root, 4 GB for swap, the rest for ZFS usr and stuff) on each disk and created two zpools with


```
zpool create zroot mirror /dev/ad4s1a /dev/ad5s1a /dev/ad6s1a
zpool create zpayload raidz1 /dev/ad4s4a /dev/ad5s4a /dev/ad6s4a
```

which worked flawlessly. Then it starts to get complicated. Mostly, I followed the article about installing on GPT RAIDZ1 (http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1), namely

- setting fletcher4 on both pools
- creating tmp, usr, var with complete hierarchy on zpayload
- cd'ing to /dist/8.0-*
- exporting DESTDIR=/zpayload, completing all steps to the kernel installation
- exporting DESTDIR=/zroot, installing the kernel and copying stuff to /zroot/boot/kernel/
- setting readonly for zpayload/var/empty
- chroot'ing to /zpayload
- creating /etc/rc.conf and /boot/loader.conf (with vfs.root.mountfrom="zfs:zroot")
- making the bootloader
- executing passwd and tzsetup, creating mail aliases
- exiting chroot
- moving /zpayload/boot to /zroot/boot
- copying /boot/zfs/zpool.cache to /zroot/boot/zfs/zpool.cache
- creating the fstab
- setting LD_LIBRARY_PATH
- unmounting filesystems

and executing


```
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zpayload/tmp
zfs set mountpoint=/usr zpayload/usr
zfs set mountpoint=/var zpayload/var
```

Rebooting the machine results in a kernel panic stating that init cannot be found when 'vfs.root.mountfrom="zfs:zroot"' is set and a prompt to select the shell and the ZFS mounted r/o when using 'vfs.root.mountfrom="zfs:zpayload"'.

Now, where is the problem?  I know that there are some additional commands to execute when using UFS for boot but then, things are a bit different with ZFS only...

Kind regards
Thomas


----------



## phoenix (Jan 15, 2010)

volatilevoid said:
			
		

> I'm trying to install FreeBSD with ZFS on a
> 
> Pentium 4 3.0 GHz
> 2 GB RAM
> ...



FreeBSD 7.2+ (possibly earlier) supports GPT partitioning on non-EFI PCs.



> so I tried to mix the howtos from http://wiki.freebsd.org/RootOnZFS to create a mirrored zpool and a RAIDZ1 one as zfsboot is only able to boot from a mirrored zpool.



FreeBSD 7.2-STABLE (ie 7-STABLE after 7.2 was released) and 8.0 support booting from raidz1 pools.

IOW, you don't need to do anything weird and wonderful and bizarre to make this work.  (Well, other than the regular hoop-jumping to get FreeBSD installed without using sysinstall.)


----------



## volatilevoid (Jan 16, 2010)

Hi phoenix,

thanks for your reply. I was wondering that Wikipedia stated that FreeBSD supports GPT on x86 so I tested it on VirtualBox amd64 with a i386 kernel. But when I was trying to boot, the BIOS gave me an


```
int13_harddisk: function 42. Can't use 64bits lba
```

error message. Because of that I tried the way stated above on the real hardware (without too much luck, but I admit it wasn't so well planned ).


----------



## volatilevoid (Jan 16, 2010)

I investigated this issue a little bit further and googling the error message revealed that the code comes from the Bochs BIOS which seems to be used by VirtualBox.

Function 0x42 from int 0x13 is _extended read_ as rombios.c from Bochs showed. There is a check whether the upper 4 bytes from the desired LBA to read are zero. If they are not, the BIOS shows an error message like the one above.

Now I wonder why there is such a value in my partition table - my test disks in VirtualBox are even smaller than 40 GB, namely 8 GB. There is someone at freebsd-questions with exactly the same problem (http://www.mail-archive.com/freebsd-questions@freebsd.org/msg227027.html) - but unfortunately without a solution.

Any suggestions? I'll try it on the real machine on Monday...


----------



## phoenix (Jan 16, 2010)

Sorry, I've never used VirtualBox to experiment with ZFS.  My VirtualBox uses have been limited to testing carp(4), lagg(4), and vlan(4).


----------



## volatilevoid (Jan 17, 2010)

No problem, phoenix.

I think I'll give it a try on Monday and see if it works for me.


----------



## volatilevoid (Jan 17, 2010)

By the way, is it enough just to install _base_ (and maybe _manpages_) like if I'd do a minimal install with sysinstall? The how-to for RAIDZ1 mentions _base_, _catpages_, _dict_, _doc_, _games_, _info_, _lib32_, _manpages_ and _ports_. I only need a minimal install as I would get the ports via portsnap after reboot.


----------



## volatilevoid (Jan 18, 2010)

I successfully installed FreeBSD 8.0-RELEASE on the machine today. To be sure I've got all stuff I might need, I did the full installation as described in the how-to.

After restarting and syncing the ports tree I got a kernel panic because I did not set my kmem and ZFS settings, but I corrected that. Next would be tuning ZFS.

The ZFS tuning guide reads



> A stable configuration with vm.kmem_size="1536M" has been reported using an unmodified 7.0-RELEASE kernel, relatively sparse drivers as required for the hardware and options KVA_PAGES=512.



Would that be enough to have a stable sytem (meaning leaving arc_size and VDEV cache size untouched)? Another proposal is



> For example it is possible to set vm.kmem_size and vm.kmem_size_max to 512M, vfs.zfs.arc_max to 160M, keeping vfs.zfs.vdev.cache.size to half its default size of 10 Megs (setting it to 5 Megs anecdotally achieves even better stability).



Setting kmem_size_max to 512M should work without modifying KVA_PAGES, shouldn't it? Currently, I'm working with the example values for 768M memory. The box has 3G now so I believe these values are very conservative. I experience some minor lags when writing many small files (e.g. when compiling a port). Would increasing kmem_size_max / arc_size / VDEV cache size help?


----------

