# 2410SA Limitation: HW RAID5 and ZFS RAID0?



## Simba7 (Jan 16, 2014)

I have bumped into this issue with an old DL100 G1 server.The Adaptec 2410SA controller does not like arrays above 1TB (controller kernel crashes even with the latest firmware), and I have 4x 640GB drives in it. So, I was thinking of doing RAID5 in hardware making 2 pairs of ~900GB arrays and striping them together in software. I can't really afford a new controller, so I have to make do with what I have. I will be using FreeBSD and ZFS.

Any comments about this idea?


----------



## trh411 (Jan 16, 2014)

Why not "bypass" the Adaptec 2410SA by configuring it for JBOD and set up your four disks directly using ZFS, since you state you will be using FreeBSD and ZFS? You could set up a single four disk RAIDZ or two mirrored pairs.


----------



## Simba7 (Jan 17, 2014)

I could, but I wanted the RAID controller to handle the heavier RAID5 load and let FreeBSD handle the lighter RAID0 load. I didn't want to overburden the P4 processor.


----------



## phoenix (Jan 17, 2014)

Simba7 said:
			
		

> I have bumped into this issue with an old DL100 G1 server.The Adaptec 2410SA controller does not like arrays above 1TB (controller kernel crashes even with the latest firmware), and I have 4x 640GB drives in it. So, I was thinking of doing RAID5 in hardware making 2 pairs of ~900GB arrays and striping them together in software. I can't really afford a new controller, so I have to make do with what I have. I will be using FreeBSD and ZFS.



RAID5 requires a minimum of 3 drives.  There's no way to make "a pair of 900 GB arrays" using only 4 disks and RAID5.

The usual way around braindead RAID controllers is to make a separate 1-disk RAID0 array for each disk attached to the controller.  Then use ZFS to create the pool using those RAID0 arrays, treating them like normal disks.

That way, the controller deals with 4 separate 640 GB RAID0 arrays, and ZFS does the rest without any size limitations anywhere.

With 4 drives, you have a couple of options (in order of throughput/IOps, slowest first):

a single raidz2 vdev using all 4 drives for 1.2 TB of usable storage
a single raidz1 vdev using all 4 drives for 1.8 TB of usable storage
2 mirror vdevs (2 disks per vdev) for 1.2 TB of usable storage
Option 1 gives you the most redundancy, allowing you to lose any 2 disks without losing data (lose a 3rd disk, pool is toast).
Option 2 gives you the most usable disk space, but you can only lose 1 disk before losing data (lose a 2nd disk, pool is toast).
Option 3 gives the same amount of space as Option 1, but gives you the most IOps and flexibility going forward.  You can lose 2 disks without losing the pool if the disks are from different vdevs.  Lost both disks in a mirror and the pool is toast.

Personally, I'd go with Option 3 (it's what I use at home).


----------



## ralphbsz (Jan 18, 2014)

Simba7 said:
			
		

> I have bumped into this issue with an old DL100 G1 server.The Adaptec 2410SA controller does not like arrays above 1TB (controller kernel crashes even with the latest firmware), and I have 4x 640GB drives in it. So, I was thinking of doing RAID5 in hardware making 2 pairs of ~900GB arrays and striping them together in software. I can't really afford a new controller, so I have to make do with what I have. I will be using FreeBSD and ZFS.



I'm not sure I understand how you want to use your RAID card.

In raw hardware, you have 4 x 640 GB = 2560 GB.  Really dumb RAID controllers can only make disk arrays out of whole disks.  With RAID-5, you always take one parity disk and use the rest for data, so this would give you a 1920 GB array (except, as you say, your controller doesn't like that).

I think what you are suggesting is that the controller is not completely dumb, and can build logical units or arrays that use parts of a disk, namely two 3+P arrays built out of 320 GB half-disks, each with 960 GB useful capacity.  You seem to be implying that your controller can handle them, because they're smaller than 1TB .  Good.  Then you want to strap those two together using ZFS, fundamentally just concatenating them logically, and ending up with 1920 GB useful capacity.  Performance will be a little weird, because ZFS will think that it has two independent volumes to work against, and when doing IO scheduling, it won't know that if you send an IO to array1, that slows down array2.  And if ZFS tries to stripe sequential data across array1 and array2, that will lead to a terrible zig-zag access pattern on the real disk drives, which would be utterly performance killing.  But all this could work.

Now, is it a good idea?  You say that you want to do the heavy lifting of parity-based RAID In the hardware, because your CPU is an old P4.  Certainly a sensible argument.  But: Small cheap PCI RAID controllers can be remarkably crappy, for example in performance.  And modern software RAID implementations have learned quite a few tricks.  

The alternative is obvious: Tell the Adaptec to get out of the way, and just present the four drives as four block devices to the OS (fundamentally, put the RAID controller into JBOD mode).  Then use ZFS to turn the four drives into a 3+P array.

You should probably benchmark both approaches, before letting your (sensible) prejudices drive you to a decision.  I would not be surprised at all if ZFS in the CPU running a sensible disk scheduler and a great cache management algorithm might outperform the Adaptec hardware, already on the initial benchmark.  Obviously the benchmark should use a workload similar to what you really do.  If your intended use is as a build machine running parallel make, and your benchmark is a sequential dd of a terabyte, you're fooling yourself (and vice versa).

What would I do?  The obvious answer is: Buy, cheat or steal yourself a modern disk controller and a modern disk, and get on with life.  The problem with this approach is that you might get caught stealing it, and you said this is not an option.  So forget this.  My serious advice would be this: If you can survive with 1280 GB of capacity, then just use the four disks in mirrored mode, using ZFS for mirroring.  If you If you absolutely need the 1920 GB of capacity, then use ZFS to do the RAID-Z 3+P array using four raw disks.  If the Adaptec hardware RAID is actually faster, and you absolutely need the extra speed, the go with your 2x 960 GB solution, 

Here's why.  I really don't like small hardware RAID controllers.  To begin with, they tend to not be terribly fast in RAID mode, in particular on small writes.  They also tend to have serious issues in case of power loss or controller firmware failure; those tend to cause data loss more often than one likes.  In contrast, ZFS is remarkable stable and safe.  Another issue is the following: The data loss rate of a RAID array depends crucially on the rebuild time of the array.  And I wonder whether after a disk failure, your P4 CPU with ZFS might be able to rebuild faster onto a spare disk, than the Adaptec can do it.  This is particularly true if your file system isn't completely full: ZFS only needs to rebuild allocated space, while the Adaptec has to rebuild every block (it doesn't know whether the space holds a file or is free space).  Furthermore, ZFS has very good scrubbing, which will help find bad sectors much earlier.  With big disks, the biggest cause of data loss is a "strip kill": You suffer one disk failure, and while rebuilding onto a spare disk, you find a bad sector on one of the good disks, and now your data is toast.  Regular scrubbing is the best prevention of strip kills.  One other argument is this: If you use ZFS to do the RAID, you can easily set up a monitoring system, which alerts you quickly when a disk fails (because the sooner you get there with a spare and start the rebuild, the less likely it is that a second disk goes down before you get there).  With a hardware RAID solution, you need to set up complex and flimsy monitoring software to accomplish the same thing.  The worst thing that can happen to a RAID array is that a disk fails, the array is running in degraded mode (waiting for a spare to rebuild onto) for weeks or months, and nobody knows about it.

Also, I don't believe that parity-based RAID codes are a good idea, either for small hardware-based controllers (parity-based RAID is actually horrendously complex, and implementations on boards tend to have holes in both performance and reliability), nor for household or amateur workloads (which tend to be heavy on small writes, and bursty and uneven workloads).  RAID-5 is great for big sequential stuff, like data mining or video processing, or supercomputers.  It should be left to highly trained experts in white lab coats (and with a good liability insurance policy).  But your workload is probably not that, nor do you look good in a white lab coat.

Hope this helps.  And if it doesn't, it might at least amuse you.


----------



## Simba7 (Jan 18, 2014)

Thanks, @ralphbsz. Finally, someone who figured out what I was trying to do.

The controller is a picky little SOB, and will not allow me to put the disks in JBOD mode unless I do Volume1, Volume2, etc. Also, the controller will not let me utilize smartmon on the drives, so that's another black mark on the controller. Not to mention it loves to crash (the controller's kernel) if you create any array >1TB. I knew some Adaptec controllers were crap, but this is amazingly craptacular.

What I eventually did, and found at a rather good price ($12, eBay), was a 3ware 9500S-LP4 card. I have utilized these in a few other servers and they work remarkably well. Not to mention, they play nice in FreeBSD.

Thanks for the ideas, though.. and figuring out how ZFS works with RAID volumes.


----------



## ralphbsz (Jan 18, 2014)

Smartmon is another good ingredient to keep your disks happy.  Forgot to mention that.  If you start seeing recoverable errors creep up on a drive, it's time to have the spare drive ready.  And if a drive reports via SMART that it is about to die, then it might be a good time to listen to it.  So a RAID controller that gets between you and running smartmon (or prevents you from scrubbing the drive in software, like ZFS does) is a bad thing.

I've been very impressed with how easy to use ZFS is.  Once you learn the language (volumes, pools), it's pretty obvious.  I ended up setting up a throw-away system (fresh FreeBSD install on random hardware), and spending a few hours playing with ZFS, to learn the ropes.  Then I set up my production home server, and everything works like a charm.


----------

