# Problem with ZFS boot on hardware RAID



## cinyar (Oct 2, 2010)

Hi 
I have a problem running full ZFS on freebsd-8.1 amd64

my hardware is an IBM x3200m3 with ServeRAID M5015 controller (with 4x2TB IBM drives in RAID5 array but I don't think that matters). 
I tried several times setting up ZFS boot (according to the documentation and different guides around the net - or using mfsBSD zfsinstall script). If I use UFS2 there is no problem. But with the ZFS boot - it looks like the raid isn't bootable. there is no error message or anything, the boot process(of the server) just skips the booting from the controller and goes to network boot. I can't really figure out where the problem is. I know I can use UFS2 boot and the rest for ZFS but I'd rather avoid this. Any idea would be helpfull. thanks


----------



## gkontos (Oct 2, 2010)

How exactly do you make your disks bootable ?

George


----------



## cinyar (Oct 2, 2010)

gkontos said:
			
		

> How exactly do you make your disks bootable ?
> 
> George



using

[CMD=]gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0[/CMD]


----------



## gkontos (Oct 2, 2010)

cinyar said:
			
		

> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0
> 
> sorry


Is mfid0 your raid card device ?


----------



## cinyar (Oct 2, 2010)

gkontos said:
			
		

> Is mfid0 your raid card device ?



yes and the 1st partition has the freebsd-boot FS type


----------



## gkontos (Oct 2, 2010)

cinyar said:
			
		

> yes and the 1st partition has the freebsd-boot FS type



You should actually use the card as JBOD and install directly to your hard disks. Then let ZFS handle the raid.


----------



## cinyar (Oct 2, 2010)

Ok, a little update here. I found out that the problem is not with ZFS but with GPT. For some reason even if I installed FreeBSD using this guide and it still won't boot. I'm going to try to upgrade the server firmware and see if it helps. With normal MBR scheme everything works fine. The problem here is that my RAID has a total storage space of 5.5T and MBR has problems with partitions larger than 1.5T. Hope I will solve it somehow.


----------



## phoenix (Oct 3, 2010)

Ideally, you would configure the RAID controller to create "single drive" array, or to use JBOD (just a bunch of disk) mode.  Then use ZFS to create the redundancy via mirror or raidz vdevs.


----------



## jem (Oct 4, 2010)

It's possible your system is skipping booting from the hardware RAID because it can not locate an active primary MBR partition on the RAID volume.  Some BIOS's check for this.

GPT creates a protective MBR with a single partition entry spanning the entire disk, but it doesn't flag this partition as active.

You can possibly use`# fdisk -a -1 /dev/mfid0` to fix that, but it _might_ break your primary GPT table. fdisk() is for working with real MBR's, not GPT's PMBR.  It has worked for me though, although it complains a bit.

The above comments about switching to JBOD and letting ZFS look after the disks, while they have merit, are a bit of a digression in my opinion.


----------



## cinyar (Oct 4, 2010)

jem said:
			
		

> It's possible your system is skipping booting from the hardware RAID because it can not locate an active primary MBR partition on the RAID volume.  Some BIOS's check for this.
> 
> GPT creates a protective MBR with a single partition entry spanning the entire disk, but it doesn't flag this partition as active.
> 
> ...



I actually followed the gpt guide on your site - didn't work. I will try again with the fdisk at the end. I know that the JBOD solution would work - letting ZFS take care of the mirroing. But that would render my not so cheap raid card useless . Thanks for the tip I'm going to try again and hopefully it will work.


----------



## cinyar (Oct 4, 2010)

Well it doesn't seem to work. But I was thinking. Would it be possible to use some usb-stick/cd to store only the bootloader and have root on the RAID array? Or is it just plain stupid and I should go with the JBOD solution?


----------



## jem (Oct 4, 2010)

Have you tried testing a basic GPT+UFS setup?  It might help narrow down the problem to either GPT or ZFS.

Just create a bunch of freebsd-ufs partitions for /, /var, /usr and write the gptboot image to the freebsd-boot partition instead.


A couple of ZFS things to double-check though (things I often forget):
- Is the bootfs property correctly set on the pool?
- Was /boot/zfs/zpool.cache file copied from the Fixit environment into the ZFS filesystem before finishing the setup?


----------



## cinyar (Oct 5, 2010)

as As I said, I followed the guide for GPT+UFS from your notes - it didn't work. for For some reason my IBM ServeRAID M5015 can't boot FreeBSD from GPT. 

I will try linux to see if it can't boot GPT at all or if it's FreeBSD specific problem. And, yes, I checked all the settings on the card to see if it's not misconfigured.


----------



## cinyar (Oct 5, 2010)

BTW would it be possible to use grub2 instead of the FreeBSD loader?


----------



## blackhawk (Mar 5, 2011)

I have same problem. 
How to solve the problem?


----------



## gessel (Jun 14, 2011)

*Same issue*

Same configuration - IBM with M5015 RAID card, same result - boots to "boot failure". I've tried this https://www.dan.me.uk/blog/2010/02/08/booting-from-zfs-raid0156-in-freebsd/ and variations like 
http://www.disi.me/index.php?id=freebsd-82-zfsnfs and http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2 to no avail. The result is the same.  

The reason for using the battery-backed RAID card instead of letting ZFS control the RAID configuration (I've installed ZFS successfully on JBOD systems) is that the RAID card does good things in hardware and has battery-backed write cache.

If we figure out a solution, we'll post it: if anyone else has any hints - they'd be much appreciated.


----------

