# New System - Software RAID5 Questions



## Ruler2112 (Dec 10, 2010)

I am putting together a new work PC for myself to use out of an old server.  It has a dual core P4 2.8 ghtz with 4 gig of ECC RAM and currently 2 IDE DVD-ROM drives.  (Quite a step up from the rather sickly AMD1800+ I'm currently on!  )  My plan is to replace one with a DVD burner.  I pulled the single 120 gig SATA drive (with windoze server on - ACK! x( ) and replaced it with 3, 1 TB SATA drives.  I'd like to use RAID5 for both speed and (more-so) reliability, but after researching things, have questions and concerns.  I store disk images and work data on this machine, so reliability is more important than speed.  However, I am going to be using X as a desktop on it and will be running Firefox, Thunderbird, OpenOffice, GIMP, VNC/RDP, burning software, and console windows to do coding w/vim, so I'd like it to be reasonably fast.

I've done mirroring on a couple of servers before with gmirror - worked very simply and effectively.  Went through the install process, then followed instructions to create the RAID-1 before rebooting, and all was well.  I haven't found anything that relates to setting up a RAID5 during the install process itself, only a few describing how to add disks to a system that's already up and running.  This will be the first BSD system I've set up with RAID5, though I have done it on Linux; the slackware box I'm running on currently in fact has RAID5.

I've read that the boot volume in FreeBSD cannot be on a RAID5 as in Linux.  Because of this, my tentative plan is to use gmirror to mirror/RAID1 the boot/root volume between 2 of the drives, then make a partition of the same size on the third drive to use for something like /var/ports, /tmp, or some other file system that can be easily re-constructed if it were to be lost.  (This would effectively give me RAID5 on the boot/root volume as no matter which of the 3 drives failed, I'd still be OK.)  I'd then have a 3 gig partition on each drive for swap so that FreeBSD can load-balance the swapping tasks between all 3 drives.  The remainder of each drive would be added to the RAID5 and be used for the rest of the system - /etc, /home, /usr, and so on.

I also can't seem to find any consensus on which software is the most reliable, stable, fast, etc for this task.  Some recommend using ZFS, others have tales of horror having used ZFS.  Others say that vinum/gvinum is the way to go, others can't seem to get it to work.  Still other people insist that geom raid 5 is the way to go, even though it's not included in FreeBSD by default.  The only option that the handbook mentions is vinum, but I don't know whether I should interpret this as an endorsement/good thing or not.  This seems to be a much-talked about topic, but everybody has their own opinions, all of which are different, and everybody is 100% positive that their own opinions are correct.  It's almost like religion...  

I'm planning on installing FreeBSD 8.1 (the latest release) on this system unless there are compelling reasons presented not to.  I'm thoroughly sick of the binary upgrade system and dependency hell in Linux, which is one of the primary reasons I'm making the switch on my main PC to BSD.  If I want to update Firefox/Thunderbird, I'd have to update major chunks of the system, development libraries, etc; the last time I tried to do this, I nuked the box.  From what I've read, this is not all that uncommon.  Therefore, I'm still using Slackware 10.2, Firefox 2.0.20, Thunderbird 1.5.9, etc.  Building software from ports is easy and relatively quick, so I'm hoping I'll be able to stay reasonably current in the software I use without having to update the entire OS every few months.  (I really don't have the time or willpower to do this without a good reason, which is why I'm still using Slack 10.2.  From the time I arrive at work, I'm being pulled in 10 different directions at once and the last thing I want to think about is upgrading the OS on and possibly killing my box.)



My questions:

1.  The BIOS has 3 modes for the SATA drives - IDE, RAID, and AHCI.  Which would be the most appropriate to use in this situation?  Since I'm using software RAID, I was thinking IDE.  Not familiar with AHCI mode though.  The only time I've changed a setting like this is on a board with hardware-RAID built in, which I do not believe this board has.  (There are no additional options, no matter what I set the mode to.)

2.  What software package would be best to use for reliability first, then speed?

3.  Does anybody know of a tutorial, how-to, or walk-through on how to set up the software in question 2, particularly during the initial system installation?

4.  Is there a guide to whatever performance tuning options exist for the software in question 2?  How about general numbers for optimal performance under most conditions?

5.  Does my plan to use gmirror for the boot volume and a partition of the same size on the third disk make sense or am I missing something?

6.  Will pretty much any PCI sound card work with BSD?  Because it's a server, there's no sound, which is something I'd very much like to have.

7.  Are there any other questions that I should be asking or considerations I should be thinking about this?


----------



## phoenix (Dec 10, 2010)

Set the BIOS to AHCI.

Download the PC-BSD 9 ISO.  Boot from it.  Use it to configure the drives into ZFS pool using a single 3-disk raidz1 vdev.

Then, either install PC-BSD so you get a working KDE desktop out-of-the-box.  Or, select the FreeBSD install option to get a plain FreeBSD install, from which you can install X, etc via ports.

Done.  

This is similar to my home setup, although that uses a 2 GB USB stick with UFS to boot, with multiple raidz1 vdevs for everything else.  It was done this way as I started with ZFS back in the FreeBSD 7.x days before you could boot from it.


----------



## Ruler2112 (Dec 10, 2010)

phoenix said:
			
		

> Set the BIOS to AHCI.
> 
> Download the PC-BSD 9 ISO.  Boot from it.  Use it to configure the drives into ZFS pool using a single 3-disk raidz1 vdev.
> 
> ...



I assume you meant PC BSD 8.1?  It's the latest I can see on their site.  I downloaded FreeBSD 8.1 last night, but will start PC BSD downloading just before I leave tonight.  (If I download an ISO during work hours, it tends to make everybody else mad...  )

Can BSD now boot from a RAID5 drive?  According to what I found in the handbook, this isn't possible.  However, the only RAID software in the handbook is vinum.  This is probably a stupid question given that it's in BSD, but is ZFS reliable?

Your described method does sound much simpler than I thought it would be.  I'm used to installing, configuring, and tweaking everything by hand - it'll be nice to not have to do this.  


One other thing - there are currently 2 IDE DVD-ROM drives in this box.  I need to replace one with a DVD burner.  Would it be better to get an IDE burner or use my last SATA channel and get a SATA burner?  I'd like to get an IDE and save the SATA port, but my experience with Linux and the ide-scsi emulation layer has not been good.  I know BSD is a different animal, but wanted to get the opinions of more knowledgeable people before buying one.  Still have to get a sound card too...


----------



## phoenix (Dec 10, 2010)

Ruler2112 said:
			
		

> I assume you meant PC BSD 8.1?  It's the latest I can see on their site.



Thought they just announced a snapshot ISO for PC-BSD 9.0?  Either way, PC-BSD 8.1 installer will do what you need.



> Can BSD now boot from a RAID5 drive?



FreeBSD has always been able to boot from a RAID array, whether it be a true hardware RAID (like a 3Ware, or Adaptec, or Areca controller), or a pseudo-RAID/fakeRAID (like a Highpoint, or Intel, or nVidia onboard controller), or a software RAID (using gmirror(8) or vinum() or gvinum().

FreeBSD 7.x was able to boot from a ZFS pool comprised of a single mirror vdev (aka RAID1).

And FreeBSD 8.x is now able to boot from a ZFS pool comprised of a single raidz vdev (aka RAID5).



> This is probably a stupid question given that it's in BSD, but is ZFS reliable?



Define "reliable".    Many people use it on their home systems.  Many people use it on work systems.  Many people use it without any issues.  Some people run into issues.  It really depends on the amount of RAM in the machine, running the amd64 (64-bit) version of FreeBSD, and taking the time to tune things a bit.

My home machine locks up on a regular basis, either from overheating or from running out of RAM (only using the 32-bit version with 2 GB of RAM).  Haven't lost any documents, movies, apps, yet.  And haven't had to suffer through a long fsck since switching to ZFS (/ is a 2 GB USB stick, so the fsck on that is virtually non-existant).

Our backups servers at work are using ZFS.  Other than some hiccups with FreeBSD 7.0, 7.1, and 7.2, and some mixups in the pool configuration in the beginning, things have been super-stable for us.  127 remote systems backed up every night, with daily snapshots going back 6 months, 1 snapshot per week for another 3 months, and 1 snapshot per month going back 3-6 months, depending on disk space.



> Your described method does sound much simpler than I thought it would be.  I'm used to installing, configuring, and tweaking everything by hand - it'll be nice to not have to do this.



There's work on porting the PC-BSD installer (pc-sysinstall) over as a replacement for the FreeBSD installer (sysinstall), so it will soon be simple for everyone to get a FreeBSD-based system up and running on top of ZFS.    Hopefully in time for FreeBSD 9.0.  



> One other thing - there are currently 2 IDE DVD-ROM drives in this box.  I need to replace one with a DVD burner.  Would it be better to get an IDE burner or use my last SATA channel and get a SATA burner?



I've only ever use IDE/ATAPI burners, so can't comment on how they compare to SATA burners.  No problems with IDE burners so far.



> I'd like to get an IDE and save the SATA port, but my experience with Linux and the ide-scsi emulation layer has not been good.  I know BSD is a different animal, but wanted to get the opinions of more knowledgeable people before buying one.



The only caveat to using an IDE burner is that K3B (the KDE-based burning software) will only work when using atapicam (the "old way" of accessing IDE devices via SCSI emulation).  It will not work with *ATA_CAM* enabled/compiled into a custom kernel (the "new way").



> Still have to get a sound card too...



Any PCI-based soundcard should work, especially if it's at least a year old.


----------



## Ruler2112 (Dec 21, 2010)

Thanks for your help Phoenix.  I have a couple more questions though, if you don't mind.




			
				phoenix said:
			
		

> Download the PC-BSD 9 ISO. Boot from it. Use it to configure the drives into ZFS pool using a single 3-disk raidz1 vdev.
> ...
> FreeBSD 8.x is now able to boot from a ZFS pool comprised of a single raidz vdev (aka RAID5).



I've booted from the PC-BSD DVD and told it to install FreeBSD.  I figure that while it'll take a while to install KDE from ports, I don't want to have to go elsewhere for help - you guys are great here.   I then got to the disk setup and am unsure of how to proceed.  How does one create a ZFS pool encompassing more than one disk?

I've tried checking additional disks when setting up the first, but don't know whether to select ad8 or ad8s1.  (My 3 SATA hard drives show up as ad8, ad9, and ad10.)  No matter which I pick, after I hit save and go back to the main setup screen and select one of the other disks (that I checked earlier), it says all the space is unused. :q

I also tried creating a raidz on each drive, but then I have 3 items listed, which I'm thinking is wrong because you said to create a single pool.  I'm thinking that I'm missing a major piece here because I haven't done anything with a vdev yet...




			
				phoenix said:
			
		

> Define "reliable".    Many people use it on their home systems.  Many people use it on work systems.  Many people use it without any issues.  Some people run into issues.
> ...
> My home machine locks up on a regular basis, either from overheating or from running out of RAM (only using the 32-bit version with 2 GB of RAM).  Haven't lost any documents, movies, apps, yet.



Not losing files is good, but locking up regularly is not.  (That's almost like windoze! x( )  Overheating isn't the OSs fault, but I don't want to have a flaky system.  I have 4 gig of RAM in this system and downloaded the x86 version of PC-BSD 8.1 - would one of the other options, gvinum or geom raid 5, be a better solution?




			
				phoenix said:
			
		

> The only caveat to using an IDE burner is that K3B (the KDE-based burning software) will only work when using atapicam (the "old way" of accessing IDE devices via SCSI emulation).  It will not work with *ATA_CAM* enabled/compiled into a custom kernel (the "new way").



I got a 24x ASUS SATA burner, so no more drive space for me unless I replace it.  Not only do I not like the idea of having to play to get K3B working, how can ya say no to a DVD-+RW for $17 shipped?




			
				phoenix said:
			
		

> Any PCI-based soundcard should work, especially if it's at least a year old.



I got an old Soundblaster Live with a CT4870 chip.  The guy wasn't sure if it worked or not, but I guess I'll find out!


----------



## phoenix (Dec 21, 2010)

Ruler2112 said:
			
		

> I've booted from the PC-BSD DVD and told it to install FreeBSD.  I figure that while it'll take a while to install KDE from ports, I don't want to have to go elsewhere for help - you guys are great here.   I then got to the disk setup and am unsure of how to proceed.  How does one create a ZFS pool encompassing more than one disk?



Unfortunately, that I can't help with.  I've never done a complete install onto ZFS (all my ZFS systems use UFS for / and /usr), just gone through the motions a few times in VMs using the PC-BSD installer.



> Not losing files is good, but locking up regularly is not.  (That's almost like windoze! x( )  Overheating isn't the OSs fault, but I don't want to have a flaky system.



Who said this had anything to do with ZFS or the OS?  It's flakey hardware for sure (bottom of the barrel P4 system, 32-bit OS install, only 2 GB of RAM, doing tonnes of file I/O and CPU intensive stuff, sending files over the network pretty much 24/7).  However, because I use ZFS on this crappy system ... I've never lost any files.  



> I have 4 gig of RAM in this system and downloaded the x86 version of PC-BSD 8.1



Unless this is a desktop system that needs Flash, you should be running the 64-bit version.



> - would one of the other options, gvinum or geom raid 5, be a better solution?



gvinum is a pain to configure correctly.  And graid5 isn't officially integrated into FreeBSD yet.



> I got a 24x ASUS SATA burner, so no more drive space for me unless I replace it.  Not only do I not like the idea of having to play to get K3B working, how can ya say no to a DVD-+RW for $17 shipped?



SATA and ATA are not the same thing.  Chances are, your system will support AHCI on the SATA controller (check the BIOS), so you can use the ahci(4) driver, and everything with Just Work.

And, if the BIOS doesn't support AHCI-mode, then you use the normal ata(4) driver and everything Just Works.

All I was saying is that if you compile a new, custom kernel, don't use the *ATA_CAM* option, as that will break K3B.


----------



## DutchDaemon (Dec 21, 2010)

(Flash runs fine on amd64 ..)


----------



## Ruler2112 (Jan 6, 2011)

I've so far been unable to figure out how to get ZFS to work with the PC-BSD installer.  Searching online produces questions asking basically the same thing as I am, but no solutions.  Hence, I have developed a new plan.

Since I have 3 drives, I'm going to install BSD / to a 25 gig UFS+J partition on one of them, then use the directions I've used in the past to get it mirrored to an identically sized partition on the second drive after installing.  I'm going to have a swap partition on each drive of about 3-4 gig so the system can distribute the swap load over all 3 drives.  I'm then going to set up ZFS manually (the process for which I _have_ found information on) and set it up as /newhome.  Once I'm satisfied that everything works as it should, I'm going to copy the contents of /home to /newhome, update my fstab, rename /home to /oldhome, and mount the ZFS partition as /home.  From what I can see, this should allow me to have redundancy and speed both.  I am going to have a leftover 25 gig partition on the third drive, but figure I can either use this for a tertiary root fs backup and/or graft it to a non-vital branch of the file system later.  Anybody see any potential problems with this strategy?

I'm currently downloading the amd64 version of FreeBSD 8.1.  Here's an insanely stupid question - will the AMD64 version work with the Intel 64 bit chip that's in this box?

One other thing, this box only has an onboard graphics card built into the motherboard.  I was thinking of getting a PCI Express video card to speed things up, but there is a problem with this.  The board has a PCIe x4 slot in it and I cannot find a video card other than x16 or x1.  There is also one extra PCI slot and a PCI-X slot.  What would be the best route to go, or would the increase in speed be negligible?

Same type of question - would the increase in speed going from 4 gig of RAM to 8 gig be noticeable or worth the $200 replacement sticks would cost?  The speed could also go from the PC5300 sticks I have now to PC6400 without a change in cost.

Thanks for the info about flash working on amd64 DD; I was ready to use graid5 on the 32-bit OS before reading that.  I've found references online to ZFS being unstable/locking up unless used under a 64-bit OS.


----------



## cteam (Jan 9, 2011)

Sorry to toss my two cents in late.  If your going to roll zfs, i would recommend checking this out:
http://mfsbsd.vx.sk/

I had trouble locating the 8.1-RELEASE tarball, but I solved that problem by just copying the BSD tarball onto a usb stick, mounting it, then installing with instructions provided.  Very simple way to get rolling on zfs.  However, since I run a zfs mirror, I was unable to set up booting from both drives at install time.  But if your not going to mess with a mirror setup you should be fine, only a matter of a few commands.

Dont worry about zfs stability, its been fine for me - amd64, 4gigs ram.  Some people have reported problems in the past using rtorrent on a zfs machine (as It hogged i/o and ram... causing bad things to happen).  It may be worth waiting a while untill 8.2-release is out, that way you get zpool v15 right off the bat without having to go through an upgrade process.


----------

