# Build Q's on ZFS RAID-Z FreeNAS/FreeBSD server



## thenerb (Nov 1, 2010)

I've been throwing ideas around on the best way to put together my new HTPC and home file system setup. 

I'm starting to lean towards building a FreeNAS/FreeBSD box to put on my network and using a more simple head for the HTPC rather than doing an all-in-one server/HTPC combo like originally planned. 

For the FreeNAS/FreeBSD box, everything I've read is really leaning me towards using ZFS with a RAIDZ or RAIDZ2 setup. 

I realize that I should use a 64-bit CPU, and it sounds like the atom chips aren't the best choice so I think I'll go with AMD. 

Here's what I'm thinking for a mobo: ASUS M4A785TD-V EVO. $80 and has DDR3, onboard GPU for setup, 5 Sata ports, 2 PCI-e x16 ports (1 @ x16 and 1 @ x4) which allows for easy addition of SATA cards down the road for expansion. 
http://www.newegg.com/Product/Prod [...] 6813131398 

What sort or processing power do I need? Will this cut it? AMD Athlon II X2 210e Regor 2.6GHz? The TDP is only 45w. Any better options for the CPU or is this a good choice? Energy efficiency is important, but I don't want to sacrifice lots of performance. It would be great to come as close as I can to maxing out the Gigabit LAN. 
http://www.newegg.com/Product/Prod [...] 6819103908 

For memory I really don't know how much I need or what type I should get. Will I get a benefit with ECC memory? Is that even supported with this board and CPU? Does the memory clock speed make much of a difference in a FreeNAS box? Any reccomendations? 

For HDDs, I'm planning on using the Samsung Spinpoint F4 2TB HD204UI's. I can get them for $90 @ Newegg right now and have read good things about using these drives for RAID. 
http://www.newegg.com/Product/Prod [...] pk=hd204UI 

What do you all think of these components for this purpose? Does ZFS RAIDZ seem like a smart choice for a home NAS for all of the home media? 

Can I ever expand the pools in RAIDZ, or should I always build them in blocks? How many max drives go in for each parity drive in RAIDZ? is it 5 total (4 + 1 parity), or is it 4 total (3 + 1 parity)? 

What are the downsides to a RAIDZ array vs RAID5?

Thanks! I'm new to this and appreciate the help.


----------



## phoenix (Nov 1, 2010)

thenerb said:
			
		

> What sort or processing power do I need? Will this cut it? AMD Athlon II X2 210e Regor 2.6GHz? The TDP is only 45w. Any better options for the CPU or is this a good choice? Energy efficiency is important, but I don't want to sacrifice lots of performance. It would be great to come as close as I can to maxing out the Gigabit LAN.
> http://www.newegg.com/Product/Prod [...] 6819103908



For a home system any dual-core CPU around 2.0 GHz is plenty.  Obviously, the more CPU cores, the better; and dual-core is the minimum as that allows you to use compression on the filesystem and still having 1 CPU for other things. 



> For memory I really don't know how much I need or what type I should get. Will I get a benefit with ECC memory? Is that even supported with this board and CPU? Does the memory clock speed make much of a difference in a FreeNAS box? Any reccomendations?



Read the manual for the motherboard to see what it supports.  Then get the most RAM that you can afford, in such a way as to allow you to use it in dual-channel mode on that motherboard (pairs of 1 GB sticks, for example).  The more RAM you can stuff into the box, the more data can be cached in the ZFS ARC, the faster things will be.  4 GB should be the starting point, not the goal.    (It is possible to use less than 4 GB of RAM with ZFS, but that's where things get really stable and useful.)



> For HDDs, I'm planning on using the Samsung Spinpoint F4 2TB HD204UI's. I can get them for $90 @ Newegg right now and have read good things about using these drives for RAID.
> http://www.newegg.com/Product/Prod [...] pk=hd204UI


Just make sure they are *NOT* the 4K Advanced Format versions.  Stay away from those if you are planning to use ZFS, especially if using raidz.  4K drives and standard ZFS don't play well together, and you have to do a bunch of manual code changes to make it start to work, but still not well.



> What do you all think of these components for this purpose? Does ZFS RAIDZ seem like a smart choice for a home NAS for all of the home media?


ZFS mirrors give you the best performance, but the least amount of disk space.  ZFS raidz1 gives you the best compromise between speed and disk space, but you can only lose 1 disk before your pool is at risk.  raidz2 is the current best for redundancy and pool safety.



> Can I ever expand the pools in RAIDZ, or should I always build them in blocks?


Expand the total storage available in the pool, yes, by adding new raidz vdevs.  Adding disks to an existing raidz vdev, though, is currently impossible.  As is removing disks from a raidz vdev, or converting between raidz types (raidz1 to raidz2, for example).



> How many max drives go in for each parity drive in RAIDZ? is it 5 total (4 + 1 parity), or is it 4 total (3 + 1 parity)?


For raidz1, it's X number of data drives + 1 parity drive.
For raidz2, it's X number of data drives + 2 parity drives.

Thus, for 5 drives, you could create a raidz2 vdev with 3 drives worth of storage space.



> What are the downsides to a RAIDZ array vs RAID5?


There are none.  All the issues with RAID-5/6 have been solved with raidz.


----------



## aragon (Nov 1, 2010)

thenerb said:
			
		

> I'm starting to lean towards building a FreeNAS/FreeBSD box to put on my network and using a more simple head for the HTPC rather than doing an all-in-one server/HTPC combo like originally planned.


Good way to go.  With a simple head you can easily build something small and fanless while you keep the bigger, noisier storage server elsewhere.



			
				thenerb said:
			
		

> I realize that I should use a 64-bit CPU, and it sounds like the atom chips aren't the best choice so I think I'll go with AMD.


Actually I think you should stick with Intel.  They usually have the best idle power and heat metrics, and a home NAS is something that'll be idle most of the time.  You're right, Atoms aren't great, but consider an entry level, 45nm, dual core Celeron instead.



			
				thenerb said:
			
		

> For memory I really don't know how much I need or what type I should get. Will I get a benefit with ECC memory? Is that even supported with this board and CPU? Does the memory clock speed make much of a difference in a FreeNAS box? Any reccomendations?


As much as you can afford.  If you can't max out the motherboard, you need to save more.   ECC is a waste, and probably unsupported on desktop boards.  Memory speed probably not too critical on a NAS - even the slowest memory is faster than ethernet I/O.



			
				phoenix said:
			
		

> Just make sure they are *NOT* the 4K Advanced Format versions.  Stay away from those if you are planning to use ZFS, especially if using raidz.  4K drives and standard ZFS don't play well together, and you have to do a bunch of manual code changes to make it start to work, but still not well.


I'm not convinced.  Granted WD EARS are crap, but other 4k "green" drives don't seem to be as bad.  Testing at http://hardforum.com/showthread.php?t=1546137 is looking interesting, and the Samsung F4 drives are looking good from what I've read so far...


----------



## phoenix (Nov 1, 2010)

aragon said:
			
		

> Actually I think you should stick with Intel.  They usually have the best idle power and heat metrics, and a home NAS is something that'll be idle most of the time.  You're right, Atoms aren't great, but consider an entry level, 45nm, dual core Celeron instead.



Good advice ... until you recommend a Celeron.    For a ZFS server, avoid Celerons at all costs.  In fact, for any kind of server, home or otherwise, avoid Celerons.  Actually, avoid Celerons at all costs, period.    You'd be better off with an Atom than a Celeron.

Core2Duo is the low-end for Intel.  Going anymore lower-end (Celeron, Pentium, Core) is not worth it.  You'll be frustrated to no end with the lack of CPU power.

However, you can get a lot more oomph for the same amount of buck, with a barely noticeable power increase, by going with an AMD setup for a home server.  Plus, you'll get graphics that aren't "one step up from pencil and paper".    Save money on the CPU (and motherboard, and RAM), so you can get more RAM.  

ZFS requires CPU oomph, especially if you add compression to any filesystems, or increase the checksum (default fletcher4, sha256 recommended to prepare for dedupe support), or add encryption anywhere.



> I'm not convinced.  Granted WD EARS are crap, but other 4k "green" drives don't seem to be as bad.  Testing at http://hardforum.com/showthread.php?t=1546137 is looking interesting, and the Samsung F4 drives are looking good from what I've read so far...



It is impossible to make a ZFS system aligned to 4K blocks without patching the source and recompiling everything from scratch.  So far, even that is only working on OpenSolaris, and is not working quite yet on FreeBSD (there's also differences of opinions on what/where to patch in the source compared to OSol).

ZFS will write variable-sized blocks between 0.5 KB and 128 KB, split up across multiple drives (raidz), thus completely blowing out your alignment and slowing things down.  This is not something that can be fixed by playing with partition offsets or ZFS recordsize settings.

For a ZFS system, you **cannot** use 4K drives.  (At least, not right now.)


----------



## aragon (Nov 2, 2010)

phoenix said:
			
		

> You'd be better off with an Atom than a Celeron.


Why?  Example, Celeron E3500 and Celeron G1101 are both _much_ faster than any Atom, and cheaper per unit processing power too.  Only (slight) edge Atom might have is power consumption, but a G1101 in a DH55TC might even match it at idle.

Having said that, after looking at the AMD Athlon II X2 210e the OP originally quoted, it looks like a good choice too if there's a good mobo available for it.




			
				phoenix said:
			
		

> It is impossible to make a ZFS system aligned to 4K blocks without patching the source and recompiling everything from scratch.


What's wrong with gnop?


----------



## phoenix (Nov 2, 2010)

Celerons lack most of the L2 cache of non-Celerons.  They also lack hyperthreading.  They also have slower FSB speeds.  They also aren't rated for SMP (although some will work in it sometimes).  For any kind of server work, even a "simple" home server, you don't want to use a hobbled CPU like a Celeron (or a Pentium, or a Core).  For the same price as a Celeron setup, you can get an AMD dual-core setup with faster RAM.

Problem with gnop is that it doesn't survive a reboot, so you have to come up with a script that will configure it *before* anything uses the drives.  Which means it won't work for a ZFS-on-root setup.  And, if it ever fails to come up at boot, you can corrupt the data on the drives by using them directly.  It's a good hack for testing and showing just how bad the non-4K-aligned accesses are.  But it's not a solution.


----------



## aragon (Nov 2, 2010)

phoenix said:
			
		

> Celerons lack most of the L2 cache of non-Celerons.  They also lack hyperthreading.  They also have slower FSB speeds.  They also aren't rated for SMP (although some will work in it sometimes).  For any kind of server work, even a "simple" home server, you don't want to use a hobbled CPU like a Celeron (or a Pentium, or a Core).  For the same price as a Celeron setup, you can get an AMD dual-core setup with faster RAM.


All true, but I don't see how it makes an Atom better than a Celeron.  Apart from Hyperthreading, the negative attributes you've associated with Celerons are true for Atoms too, and to a greater degree.  Another major downfall of Atoms which hasn't been mentioned is their limitation of 4 GB of RAM (compared to 16 GB).  Modern Celerons have two cores, high clock speeds, plenty of cache, multiple memory channels, and low power consumption.  The Celerons of today aren't crippled as badly as the old days.  I'm inclined to say that Atoms are the new Celeron.  Here's a comparison of some Intel CPUs:

http://ark.intel.com/Compare.aspx?ids=49490,48750,48504,48505,43230,43523,


----------



## Nukama (Nov 2, 2010)

*regarding ECC*

The Asus motherboard supports ECC-memory, but only *unbuffered* DDR3-RAM. Don't buy registered memory for that board. 
AMD uses an integrated memory controller with ECC-support on nearly all AM3 processors (I'm not sure, if Sempron comes with ECC-support).

If you're going to buy Intel, you've to search for an appropriate motherboard and CPU that supports ECC (expensive). 
If you're planning to utilize virtualization you can also search http://ark.intel.com for a suitable CPU that supports VT-x.

ECC-memory can correct one-bit errors, see ECC_RAM.

As aragon stated, ECC is a waste. The same is true for ZFS.


----------



## thenerb (Nov 2, 2010)

Nukama,

Are you saying ZFS is a waste?? You just mean that spending any extra $$ on ECC memory is money not well spent, right?

Looks like 4GB of DDR3 ECC Unbuffered memory is around $80. I can probably save around $20ish if I go with non-ECC memory. Should I spend the extra $20 and go with the ECC memory or is there truly no benefit?

If ECC memory will decrease the risk of data loss in any material way I don't mind spending the extra cash.


----------



## Nukama (Nov 2, 2010)

thenerb,
it depends on the requirements & opinion. 

If you're not concerned about data integrity there is no use for ZFS and ECC, you could stripe several disk together for huge storage and performance and eventually lose all data. 

I'm concerned in data integrity and my server utilize ECC memory. 
I had some hassle with defect memory in the past and my last desktop install presumably starved because of such a defect, therefore my desktop was also outfitted with one of the memory modules from the server.

Maybe I'm over-concerned about this, and ZFS could handle defect memory (unfortunately when data inside memory gets corrupted before check-summing, you're outta luck).
Trying to build a chain with a hard segment (ZFS) and some weak segments (non-ECC, bad backup scheme, faulty hardware & software) results in a chain likely to be broken. 

So you're better off with ECC. Some boards also support Chipkill Memory.

Do you intend to buy a UPS for your server and extra disks for backups?


----------



## thenerb (Nov 2, 2010)

Yes, I already have a good UPS to use with the server and have an external drive that I will use to back up the truly critical stuff (pictures, documents, etc).

What happens if data is being written or read from a ZFS RAIDZ array at the time of power loss? Is there a large risk of corruption and total loss or is ZFS usually able to compensate and heal?

Another question. If I use the newer advanced format drives with 4K sectors. Will I get better performance with an odd number of disks or an even number of disks? I've heard that odd number of disks provides faster performance. Have you heard this?


----------



## phoenix (Nov 3, 2010)

thenerb said:
			
		

> What happens if data is being written or read from a ZFS RAIDZ array at the time of power loss? Is there a large risk of corruption and total loss or is ZFS usually able to compensate and heal?



Data is either written to disk, or not written to disk, nothing in between.  Well, technically, data may be partially written to disk, but the last step in a transaction is to update the pointers to point to the new data; if that step isn't done due to power loss, then it's the same effect as none of the data being written.  In either case, the pool is safe, the data is safe.  You may "lose" up to 30s of data, but the pool will be usable.



> Another question. If I use the newer advanced format drives with 4K sectors. Will I get better performance with an odd number of disks or an even number of disks? I've heard that odd number of disks provides faster performance. Have you heard this?



*Do not use 4K drives with ZFS.*  Why is that so hard to understand?    Even the OSol and Solaris folks avoid them.


----------



## thenerb (Nov 3, 2010)

There's an option in FreeNAS to use 4K drives. It is a checkbox that can be selected when setting up the array. I've heard that everything is fine when using that checkbox. Is it not?


----------



## phoenix (Nov 3, 2010)

Unless they know something that the ZFS devs at Oracle don't, then I would say that's false advertising.  The pool can be created and used.  But it will be extremely slow compared to a pool using non-4K drives.


----------



## thenerb (Nov 3, 2010)

I could be completely off base, but I think is enables GNOP and emulates 512b sectors.

The way the industry is heading, all drives will be 4K sector drives in no time. If that checkbox in FreeNAS doesn't truly address the issue, hopefully someone can develop an update to ZFS otherwise we will all eventually be up a creek within a year or two.

Could you recommend a good 512b sector 2TB drive for under $130?


----------



## bmcclure937 (Jan 26, 2011)

phoenix said:
			
		

> Data is either written to disk, or not written to disk, nothing in between.  Well, technically, data may be partially written to disk, but the last step in a transaction is to update the pointers to point to the new data; if that step isn't done due to power loss, then it's the same effect as none of the data being written.  In either case, the pool is safe, the data is safe.  You may "lose" up to 30s of data, but the pool will be usable.
> 
> 
> 
> *Do not use 4K drives with ZFS.*  Why is that so hard to understand?    Even the OSol and Solaris folks avoid them.



Is this still true? I know a lot of work has been done to get 4K drives working in ZFS. I was specifically looking to get either the Samsung F4 or the WD20EARS drives for my FreeNAS. I know you can use GNOP (and FreeNAS has the setting built into the GUI). 

What other steps should I take to ensure my performance is optimal? Do I need to align partitions or perform any additional steps. I know it was not recommended to use the Advanced Format 4K drives back in Sept/Oct 2010 but I wanted to check if that is still the case!



			
				thenerb said:
			
		

> I could be completely off base, but I think is enables GNOP and emulates 512b sectors.
> 
> The way the industry is heading, all drives will be 4K sector drives in no time. If that checkbox in FreeNAS doesn't truly address the issue, hopefully someone can develop an update to ZFS otherwise we will all eventually be up a creek within a year or two.
> 
> Could you recommend a good 512b sector 2TB drive for under $130?



I believe this is true as well. It seems as if all affordable disks are now moving towards 4K. Hopefully it becomes easier to use these with ZFS.


----------



## Orum (Jan 27, 2011)

bmcclure937 said:
			
		

> Is this still true? I know a lot of work has been done to get 4K drives working in ZFS. I was specifically looking to get either the Samsung F4 or the WD20EARS drives for my FreeNAS. I know you can use GNOP (and FreeNAS has the setting built into the GUI).


Not sure about native 4K sectors and ZFS, but it sounds like 512B emulation on a 4K disk (WD's EARS) is the worst of both worlds.  See this thread.

There are ways you can realign so that the emulated sectors align with the real sectors, for UFS, or force it to think the disk is using 4K sectors (which is what it's actually doing anyway) via GNOP for ZFS, but they seem like a lot more trouble than they are worth when you can greatly simplify things just by using a disk that actually reports the native 4K sectors.


----------



## rusty (Jan 27, 2011)

I'd also consider reading http://webcache.googleusercontent.c...l=uk&client=firefox-a&source=www.google.co.uk

There seems to be an authentication error with the original so linking via google cache.


----------



## bmcclure937 (Jan 28, 2011)

Thank you both for the replies. I am leaning towards the Samsung Spinpoint F4 (4K Advanced Format) drives... but I am not sure if these drives *emulate* the 4K sectors or if they actually report them properly. It is a pain that the manufacturers do not make this clear. 

@rusty: thanks for the link. I already have that bookmarked from my own research. sub_mesa is an extremely helpful and brilliant person...  I appreciate all of the work he has done. He is such a vast source of knowledge and information


----------



## JoeSchmuck (Feb 6, 2011)

I have read in several places that the F4 emulates the 4K with 512 bytes.  I have seen one thread where someone was talking about a jumper to turn on/off emulation but nothing definative.  I have three of these drives on order and I guess I will find out first hand.  I suspect the person talking about the jumper was confused about the normal use of a jumper on drives (137GB boundry/Master/Slave/CS).

I'd be interested in hearing if someone has more information on the drive emulation, if it actually does exist.

-Joe


----------



## aragon (Feb 6, 2011)

The F4 drives do do emulation.  There is no way to disable it.  AFAIK all that the jumpers do is control how sector 63 is mapped.  With the right jumper position the drives will map sector 63 to a 4k boundary so that standard partition tools create the first partition on a 4k boundary.


----------



## JoeSchmuck (Feb 6, 2011)

aragon said:
			
		

> The F4 drives do do emulation.  There is no way to disable it.  AFAIK all that the jumpers do is control how sector 63 is mapped.  With the right jumper position the drives will map sector 63 to a 4k boundary so that standard partition tools create the first partition on a 4k boundary.



For the F4 (HD204UI) do you know the jumper setting to force it to the 4k boundry?  I understand the WD drive has this feature but haven't heard anyone state the actual Samsung drive jumper position.

Thanks


----------

