# Building a small, personal NAS



## cbunn (Nov 18, 2012)

I'm planning a replacement for my home file server (low-end AMD running FreeBSD with 4x1TB HDDs in a RAIDZ). I'd like this one to be smaller, more power efficient and more robust. And obviously I'd like to have more storage space. I'm looking to build around the end of the year.

So here's my plan so far:

CASE: Lian Li PC PC-Q25-B
PSU: either reuse one I have or buy a SeaSonic SS-300ET
MOBO: Intel DBS1200KP Mini ITX Server Motherboard
CPU: Intel Celeron G540 Sandy Bridge 2.5GHz
RAM: 2x 8GB Kingston ECC Unbuffered DDR3 (not sure which model exactly)
RAID: Supermicro AOC-USAS2-L8i (to be used in IT mode)
HDD: 5x 3TB WD Red (to be used in RAIDZ)

For the memory, does 1066 vs 1333 matter for a NAS? Beyond that, there are still a number of DIMMs that appear very similar but vary in price by up to $30. Should I just get the cheapest ECC memory or is there a real difference?

Right now, I have the OS running off of USB flash drives (just the root partition; the rest is mounted from ZFS), but I might grab a small SSD if I see a decent one on sale. I don't think my needs require anything special for L2ARC or ZIL. Buying enterprise SSDs for those would break the budget anyway.

From what I can gather, all of this should have good compatibility with FreeBSD. The Intel board shouldn't be a problem and the onboard Intel NICs will save any potential Realtec headaches. The Supermicro SAS card is reported here and elsewhere to work well. I can't tell if it will come with IT firmware out of the box, but I don't have a problem with flashing that myself.

Anywho, I'm curious if this set-up sounds like a good plan. It's probably a bit overkill for my needs, but I'd like to build this properly so it'll last me a long time. Thanks!


----------



## freesbies (Nov 18, 2012)

You asked about memory clock on nas.. I can tell you that a higher clock like 1600mhz is always better than 1333mhz, it makes difference cbunn, you will notice that performance on every machine. Althought you have a good sollution too, that is : changing memory timings of that particular mem ram..


----------



## ahavatar (Nov 18, 2012)

Does Intel Celeron support ECC? I guess not.


----------



## cbunn (Nov 18, 2012)

ahavatar said:
			
		

> Does Intel Celeron support ECC? I guess not.



I wondered that as well, but according to Intel's own documentation, the G540 does support ECC on a C206 board.

From http://download.intel.com/embedded/processor/prodbrief/325980.pdf


> Based on 32nm process technology and next-generation Intel microarchitecture codename Sandy Bridge, the Intel Celeron processor G540 features dual-core processing with Intel HD Graphics and Error Correcting Code (ECC) capabilities (when paired with Intel C206 chipset).


----------



## marwis (Nov 18, 2012)

cbunn said:
			
		

> CPU: Intel Celeron G540 Sandy Bridge 2.5GHz


If you're even remotely considering toying with virtual machines, e.g. headless emulators/virtualbox-ose, I'd definitely choose a CPU with more cores.

I wanted to suggest to have a look at the other CPUs that support new AES instruction set.  This would not make sense though, as they cost about four times as much.  (It is funny Celeron G540 is listed as having no ECC support.)

The setup looks very promising.  Please, do share your experience once you have it set up.  I'd be glad to see some benchmarks/bonnie++ benchmarks.


----------



## cbunn (Nov 18, 2012)

marwis said:
			
		

> If you're even remotely considering toying with virtual machines, e.g. headless emulators/virtualbox-ose, I'd definitely chose a CPU with more cores.


Nope. Not remotely interested. I occasionally use VMs on my desktop, but I don't have any desire to do that on this server. It will do a little web serving, but basically file serving is all it needs to do well.



			
				marwis said:
			
		

> I wanted to suggest to have a look at the other CPUs that support new AES instruction set.  This would not make sense though, as they cost about four time as much.  (It is funny Celeron G540 is listed as having no ECC support.)


AES support would be kinda cool, but as you said, I'd need to spend a lot more to get it for only possible use cases. Plus, I think many of those CPUs are V2 chips which are not compatible with this board. As for the G540 listed as without ECC support. It is perplexing. On the one hand, I'd like to trust the PDF from Intel stating it, but then one never knows. What other means could I have to determine the truth (apart from trial and error)?



			
				marwis said:
			
		

> The setup looks very promising.  Please, do share your experience once you have it set up.  I'd be glad to see some benchmarks/bonnie++ benchmarks.


I'll be sure to do that. I'm clearly not the first guy to see this case as ideal for this purpose, but I suppose my collection of components is unique. As it happens, the Supermicro SAS card is on sale at Newegg now for about $112 (originally $140), so I might have to jump on that. 

I'm not terribly familiar with benchmarking, but I'll definitely look into that tool.


----------



## throAU (Nov 19, 2012)

freesbies said:
			
		

> You asked about memory clock on nas.. I can tell you that a higher clock like 1600mhz is always better than 1333mhz, it makes difference cbunn, *you will notice that performance on every machine*.



Uhh...

No you won't.

The NAS is going to be IO bound waiting for disk access.  You MAY notice some small performance difference in CPU/memory bandwidth constrained apps, but a NAS?

Nope.


And yes, I'd recommend a CPU with accelerated AES (if you plan on running encryption, if not don't bother), and if you're planning to run VMs, look for one with the VT-D instructions.


----------



## cbunn (Nov 21, 2012)

throAU said:
			
		

> The NAS is going to be IO bound waiting for disk access.  You MAY notice some small performance difference in CPU/memory bandwidth constrained apps, but a NAS?
> 
> Nope.



Also, the Celeron G540 is limited to 1066MHz RAM anyway.

And if anyone is wondering, I posted the question about ECC memory with the Celeron G540 to Intel's support board and was told that ECC support depends on the chipset and not necessarily the CPU:

http://communities.intel.com/message/172902#172902


----------



## cbunn (Nov 21, 2012)

And now Newegg has dropped the price of the Seagate 3TB drive to $90, complicating my plan. These drives aren't rated for 24/7 availability, but some people have used them in NAS boxes without issue ...

http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844


----------



## marwis (Nov 22, 2012)

I looks like your plan is simple again .  The site says


> DISCONTINUED
> This item is currently out of stock and it may or may not be restocked.


----------



## cbunn (Nov 23, 2012)

Yeah, it sold out quickly. Even more so on Amazon. But now it's back on Amazon for $99 as a Lightning deal ($109 regular price). I think the WD Red is still the better long-term buy.


----------



## cbunn (Nov 24, 2012)

Several stores (Newegg, Amazon, B&H) are offering the Seagate 3TB for around $90 now. And Newegg has a deal for 20% off all server motherboards, but the one I want is temporarily out of stock. The Universe does not want me to save money on this build, it seems.


----------



## wblock@ (Nov 24, 2012)

Think of it as the universe trying to guide you away from particular choices towards others.  Maybe not for your benefit, but still...


----------



## marwis (Nov 24, 2012)

You're putting the setup together for the next couple of years.  Saving $100 would be nice but won't really make a difference in the long run.


----------



## cbunn (Nov 26, 2012)

marwis said:
			
		

> You're putting the setup together for the next couple of years.  Saving $100 would be nice but won't really make a difference in the long run.



That's what I keep telling myself. But the savings is more like $350 for five drives.


----------



## wblock@ (Nov 26, 2012)

What's your data worth?  That's a serious question.  The drives used for disposable media storage can be a different grade than for archival data.


----------



## xibo (Nov 26, 2012)

cbunn said:
			
		

> RAID: Supermicro AOC-USAS2-L8i (to be used in IT mode)



Are you sure the UIO card works with a non-supermicro board, i.e. can be used with a generic PCI-express interface?


----------



## Sebulon (Nov 26, 2012)

xibo said:
			
		

> Are you sure the UIO card works with a non-supermicro board, i.e. can be used with a generic PCI-express interface?



Yes

ItÂ´s a little tricky though, the whole board is like "upside down", with the components facing "up", instead of "down", so you have to reverse the back-plate to make it fit the screw-holes on the back of chassis. Problem solved by using a few motherboard spacers.

/Sebulon


----------



## cbunn (Nov 27, 2012)

wblock@ said:
			
		

> What's your data worth?  That's a serious question.  The drives used for disposable media storage can be a different grade than for archival data.



Well, none of the data will be irreplaceable. The bulk of the space will be taken up my music and movies, for which I retain the optical media. The really important stuff (mainly my photos), are also kept on my desktop, laptop and cloud backup (Crash Plan). So it's a mix of media and archival, but it's not mission critical by any means. I just like to over-engineer these kinds of things to remove headaches down the line and to make the system usable for as long as possible.



			
				xibo said:
			
		

> Are you sure the UIO card works with a non-supermicro board, i.e. can be used with a generic PCI-express interface?



The interface is PCIe, but the bracket is UIO. So the simplest solution is to remove the bracket and install the card without any bracket at all. But there are also aftermarket brackets that will fit onto these Supermicro boards. I don't think they include the LED holes, but I'll rarely be looking at them anyway.


----------



## _martin (Dec 5, 2012)

@cbunn: Did you buy it yet ? I'm /still/ struggling what HW to buy for a home storage, your setup seems interesting..


----------



## cbunn (Dec 8, 2012)

matoatlantis said:
			
		

> @cbunn: Did you buy it yet ? I'm /still/ struggling what HW to buy for a home storage, your setup seems interesting..



Nope, not yet. So far, I've bought the Supermicro HBA card and a couple fans, but that's about it. I've got some price alerts on the different pieces and I'm hoping for a sale here and there (which is what happened with the Supermicro card and fans). The main issue is the cost of the HDDs. If the price of those comes down a bit, I'll bite. Additionally, I didn't realize that 8GB DIMMs of ECC unbuffered RAM is so expensive. The cheapest I can find for a 16GB set is $125 at Amazon.

As I said before, I'm not in a huge rush to build. If no price drops come along, I'll probably bite the bullet in January. When that happens, I'll be sure to post here with the results.


----------



## _martin (Dec 8, 2012)

Oki, thanks


----------



## cbunn (Jan 8, 2013)

Just as an update, the last of the necessary parts (SAS breakout cables) came in the mail today, so I'm putting things together today. I hope to have some pics up soon and then start installing FreeBSD and doing some hardware tests.


----------



## cbunn (Jan 14, 2013)

So I completed the build a few days ago and have been running some tests since then (MemTest, hard drive diagnostics, etc.). All the parts I mentioned in the first post didn't change. I got Kingston ECC RAM and some SAS cables from Monoprice (~$10 per cable). I bought a Seasonic 300ET so that I can keep the old server running for now. For CPU cooling, I used the stock Intel HSF from a Core i7 that I have in my desktop system with an aftermarket cooler because it has a copper core. I forgot how much I hate those pushpins. It seems quiet enough for my purposes. I replaced the Lian Li fans with Cougar fans. They're pretty quiet and have fully sleeved cables.

For the Supermicro card, my intention was to use an aftermarket PCI bracket since the UIO one doesn't fit and I didn't want the card to come loose. Well, it would seem that the PCI bracket expects smaller screws, because I had a hard time getting them in and one sheared off, making the bracket useless. I might end up getting another and drilling it out a bit, but for now the card is going commando.

A couple of things have been annoying, mainly to do with the BIOS on the Intel S1200KP board. It takes a long time to boot, for one thing. Also, it doesn't seem to honor the setting which would remove the Intel splash screen. And during that splash screen, I've had issues getting the BIOS to acknowledge me hitting F2 or F10 on the keyboard. I have to pretend I'm doing Morse Code for it to work, it seems. It also seems that the BIOS incorrectly reports the CPU temperature. When I check it in the BIOS, it has a high reading (56C+), but when I use coretemp in FreeBSD, it's more on the order of 48-49C. I can't imagine a reason why the BIOS would heat up the chip.

Something I'm not sure about is if I should expect to see a BIOS for the Supermicro card. I thought RAID cards usually had some firmware settings between the normal BIOS and the OS loading. But I don't see anything. It would seem that the IT firmware is already loaded, because the disks appear as stand-alone drives (da0-4) with no mention of any kind of RAID setup. So shall I just leave it at that?

As for the case, it's very well-made, but has an issue with the I/O shield. Most cases are thin steel, but this thick aluminum makes installing it difficult. I felt like I was going to break it.

Having said all that, I'm pretty happy with the results so far. The case, while difficult to work in, is very attractive and small on the outside. The hard drive bays are a very nice touch. Something I didn't see mentioned elsewhere is that it has a set of Molex and SATA power inputs for them, so you don't have to worry about which connectors your PSU has more of. All the drives seem to be working just fine so far (currently running a few passes of badblocks, then checking the SMART info).

Initially, after installation, I was having issues where the prompt would hang for a few seconds after some commands and during login (sometimes long enough for Putty to time out), but that seems to have stopped.

Are there any hard drive benchmarks I should run to see what kind of performance I can expect?

Because these are 4K drives that lie and report being 512b, do I need to do that Gnop trick when creating my ZFS pool? I'm not sure if anything has changed since I last saw that.

What's the best way to transfer the data? I see sysutils/zxfer mentioned here occasionally. Is that the best method for maintaining data integrity?

I'm at work, but next chance I get at home, I'll upload some pics.


----------



## wblock@ (Jan 14, 2013)

cbunn said:
			
		

> Something I'm not sure about is if I should expect to see a BIOS for the Supermicro card. I thought RAID cards usually had some firmware settings between the normal BIOS and the OS loading. But I don't see anything. It would seem that the IT firmware is already loaded, because the disks appear as stand-alone drives (da0-4) with no mention of any kind of RAID setup. So shall I just leave it at that?



Some cards have separate management programs.  If you planned to use it this way, good enough.



> All the drives seem to be working just fine so far (currently running a few passes of badblocks, then checking the SMART info).



There's not much point to looking for a list of bad blocks with drives made in the last two decades, which handle block remapping in firmware.  Filling the drive with zeros with dd(1) has about the same effect.  Of course, all of this is just an attempt to catch drives that are going to fail early, and it doesn't always work.  I had a WD Red 1T that showed no bad blocks, yet had a severe error that prevented it from being used in a mirror and failed the short or long SMART tests from sysutils/smartmontools instantly.



> Initially, after installation, I was having issues where the prompt would hang for a few seconds after some commands and during login (sometimes long enough for Putty to time out), but that seems to have stopped.



That doesn't sound good.  Don't know what would cause it, though.



> Are there any hard drive benchmarks I should run to see what kind of performance I can expect?



benchmarks/bonnie++ for a hard to read but mostly-realistic result, diskinfo -tv for a quick and absolute best-case result.



> Because these are 4K drives that lie and report being 512b, do I need to do that Gnop trick when creating my ZFS pool?



Yes.



> What's the best way to transfer the data? I see sysutils/zxfer mentioned here occasionally. Is that the best method for maintaining data integrity?



It looks like that's either for ZFS-to-ZFS (zfs send/receive), or just uses net/rsync.  Which can be done manually, but maybe it knows the best combination of options.


----------



## cbunn (Jan 15, 2013)

wblock@ said:
			
		

> There's not much point to looking for a list of bad blocks with drives made in the last two decades, which handle block remapping in firmware.  Filling the drive with zeros with dd(1) has about the same effect.  Of course, all of this is just an attempt to catch drives that are going to fail early, and it doesn't always work.  I had a WD Red 1T that showed no bad blocks, yet had a severe error that prevented it from being used in a mirror and failed the short or long SMART tests from sysutils/smartmontools instantly.



Well, I ran a long SMART test on each drive before the badblocks tests with no errors reported. I figure writing a bunch of random data and reading it back might be better than just zeros. It's probably all the same, but I'm fine with this taking longer.




			
				wblock@ said:
			
		

> That doesn't sound good.  Don't know what would cause it, though.


From what I've seen so far, it might be a hostname thing, but it was giving me problems even before I started using bash (which puts the hostname in the prompt) and it doesn't explain why it would hang after user name before password during a login attempt. I'm still going to keep an eye out, but it hasn't been a problem since the first day. It's possible that it has to do with running from a USB flash drive, but so does the previous server which never displayed this issue.




			
				wblock@ said:
			
		

> benchmarks/bonnie++ for a hard to read but mostly-realistic result, diskinfo -tv for a quick and absolute best-case result.


Thanks! I'll look into both.




			
				wblock@ said:
			
		

> It looks like that's either for ZFS-to-ZFS (zfs send/receive), or just uses net/rsync.  Which can be done manually, but maybe it knows the best combination of options.



It looks like a wrapper for rsync with all the relevant options pre-selected. I suppose a ZFS send/receive would be the proper method, but I am more familiar with rsync, which gave it some appeal.

The nice thing about FreeBSD and ZFS is that you can pretty much set things up once and with a little automation you don't have to do any of these tasks for a while. But then I forget how to do them and need to relearn each time.


----------



## cbunn (Jan 16, 2013)

So here's something slightly odd. As I'm doing the badblocks passes (just to test the drives, not for marking bad sectors) I can see that one drive finished hours ahead of the others, three are pretty closely timed and then one is hours behind the pack. Seems strange. The only correlation I have is that the fast drive is alone on a breakout cable from the second SFF-8087 port. I would figure that this wouldn't matter and each of the 8 ports on the controller card would have separate data lanes. When they all finish, I'll be able to do some speed tests to get some hard numbers.


----------



## cbunn (Jan 18, 2013)

So in order to get a baseline of my hard drive performance before putting the drives in a RAIDZ, I tested each drive individually using benchmarks/bonnie++. Before running the tests, I formatted and mounted them as such:

```
# dd if=/dev/zero of=/dev/da0 count=1k
# gpart create -s gpt da0
# gpart add -a 1m -t freebsd-ufs -l disk0 da0
# newfs -U -f 4096 /dev/da0p1
# mount -t ufs /dev/da0p1 /mnt.0
```

And here's the output of bonnie++ with each command being in the form of `$ bonnie++ -m disk0 -d /mnt.0`






So it looks like disk performance is on the order of 135-145MB/s. That seems okay to me. I've seen some reviews with HDTune getting just over 150MB/s out of these, but that might not be as realistic of a result. Unless I'm missing something.


----------



## wblock@ (Jan 19, 2013)

bonnie++ is going through the filesystem.  Don't know about HDTune, but a lot of benchmarks just read or write blocks on the drive.  Try diskinfo -tv ada0.


----------



## cbunn (Jan 19, 2013)

wblock@ said:
			
		

> bonnie++ is going through the filesystem.  Don't know about HDTune, but a lot of benchmarks just read or write blocks on the drive.  Try diskinfo -tv ada0.



I ran diskinfo first, actually, just to get a rough idea. And the results are on par with what bonnie++ showed.

Here's some trimmed output:

```
#  diskinfo -vt /dev/da0
/dev/da0
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0989515    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.106308 sec =   32.425 msec
        Half stroke:      250 iter in   5.677708 sec =   22.711 msec
        Quarter stroke:   500 iter in   9.435993 sec =   18.872 msec
        Short forward:    400 iter in   3.139615 sec =    7.849 msec
        Short backward:   400 iter in   2.137566 sec =    5.344 msec
        Seq outer:       2048 iter in   0.168892 sec =    0.082 msec
        Seq inner:       2048 iter in   0.159148 sec =    0.078 msec
Transfer rates:
        outside:       102400 kbytes in   0.797209 sec =   128448 kbytes/sec
        middle:        102400 kbytes in   0.820908 sec =   124740 kbytes/sec
        inside:        102400 kbytes in   1.487073 sec =    68860 kbytes/sec

# diskinfo -vt /dev/da1
...
Transfer rates:
        outside:       102400 kbytes in   0.769086 sec =   133145 kbytes/sec
        middle:        102400 kbytes in   0.871308 sec =   117524 kbytes/sec
        inside:        102400 kbytes in   1.414511 sec =    72393 kbytes/sec


# diskinfo -vt /dev/da2
...
Transfer rates:
        outside:       102400 kbytes in   0.692566 sec =   147856 kbytes/sec
        middle:        102400 kbytes in   0.893075 sec =   114660 kbytes/sec
        inside:        102400 kbytes in   1.587866 sec =    64489 kbytes/sec

# diskinfo -vt /dev/da3
...
Transfer rates:
        outside:       102400 kbytes in   0.802362 sec =   127623 kbytes/sec
        middle:        102400 kbytes in   0.843734 sec =   121365 kbytes/sec
        inside:        102400 kbytes in   1.547642 sec =    66165 kbytes/sec


# diskinfo -vt /dev/da4
...
Transfer rates:
        outside:       102400 kbytes in   0.715039 sec =   143209 kbytes/sec
        middle:        102400 kbytes in   0.816365 sec =   125434 kbytes/sec
        inside:        102400 kbytes in   1.467947 sec =    69757 kbytes/sec
```

Any idea what might explain the 20MB/s difference between some of the drives?


----------



## wblock@ (Jan 19, 2013)

cbunn said:
			
		

> Any idea what might explain the 20MB/s difference between some of the drives?



I remember seeing some variations between the last few drives I set up, but can't remember what magnitude.  Some of the WD drives claim to be variable speed, some are 5400, some are 7200.  But those may be nominal speeds, maybe the motors aren't locked to an accurate speed.  So a slightly faster motor would get the data past the heads faster.  The only other thing I can think of is a variation in actuator arm mass or magnet strength, which would make for different seek times.  Seems like the manufacturing tolerances would be tight enough to minimize that.

It would be nice to have a contact at one of the drive manufacturers to ask about this type of thing.


----------



## Terry_Kennedy (Jan 19, 2013)

wblock@ said:
			
		

> Some of the WD drives claim to be variable speed, some are 5400, some are 7200.


I don't think WD ever claimed variable spindle speed - that's something that some early reviewers came up with when they saw "IntelliPower" in the marketing literature. I don't know why WD persists in trying to hide the spindle speed - it can be inferred from benchmarks, and can be determined exactly by putting a frequency counter on the spindle motor leads, which are exposed on the underside of the drive.



> But those may be nominal speeds, maybe the motors aren't locked to an accurate speed.  So a slightly faster motor would get the data past the heads faster.  The only other thing I can think of is a variation in actuator arm mass or magnet strength, which would make for different seek times.  Seems like the manufacturing tolerances would be tight enough to minimize that.


Both the spindle and actuator motors are multi-phase, so they operate at exactly the speed the logic tells them to. Having more than a faction of a percent of "slop" would make the data separator design and servo stuff incredibly complex, not to mention data corruption from write splice errors.



> It would be nice to have a contact at one of the drive manufacturers to ask about this type of thing.


If there's a specific WD model / firmware that exhibits large variation between drives, I can ask my contact at WD about it. My initial guess is that the drives are not identical - they either have a different model suffix (on WD drives, the -xxxxxx part of WDmmmmoooo-xxxxxx) or different firmware. At various times, WD has changed the bit density on a drive model in order to reduce the number of platters / heads. There have also been cases where the zone layout was changed to improve performance.

It's also possible that the drives are performing some sort of internal error recovery. Some vendors (not to name names, but I'm talking about you, N****g) feel that putting a naked drive with only an anti-static bag for protection in a large box and throwing in a little packing material is adequate "protection" during shipping. This should show up on something like HD Tune's access time graph.


----------



## cbunn (Jan 20, 2013)

Terry_Kennedy said:
			
		

> If there's a specific WD model / firmware that exhibits large variation between drives, I can ask my contact at WD about it. My initial guess is that the drives are not identical - they either have a different model suffix (on WD drives, the -xxxxxx part of WDmmmmoooo-xxxxxx) or different firmware. At various times, WD has changed the bit density on a drive model in order to reduce the number of platters / heads. There have also been cases where the zone layout was changed to improve performance.



Well, it would seem that all five of my drives have exactly the same model suffix:


```
$ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Device Model"; done
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
```



			
				Terry_Kennedy said:
			
		

> It's also possible that the drives are performing some sort of internal error recovery. Some vendors (not to name names, but I'm talking about you, N****g) feel that putting a naked drive with only an anti-static bag for protection in a large box and throwing in a little packing material is adequate "protection" during shipping. This should show up on something like HD Tune's access time graph.



I ordered my drives from Amazon and they were properly packed. Each drive was protected by the molded plastic end caps within individual cardboard boxes, lined up inside a larger box with airbags to pad any extra space.

HD Tune is a Windows program only, right? I figured diskinfo and bonnie++ would have been the equivalent of that. Is access time the same as seek time? diskinfo reports that data. Here's the full output of diskinfo for each of my drives, in case it helps:


```
$ sudo diskinfo -vt /dev/da0
/dev/da0
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0989515    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.106308 sec =   32.425 msec
        Half stroke:      250 iter in   5.677708 sec =   22.711 msec
        Quarter stroke:   500 iter in   9.435993 sec =   18.872 msec
        Short forward:    400 iter in   3.139615 sec =    7.849 msec
        Short backward:   400 iter in   2.137566 sec =    5.344 msec
        Seq outer:       2048 iter in   0.168892 sec =    0.082 msec
        Seq inner:       2048 iter in   0.159148 sec =    0.078 msec
Transfer rates:
        outside:       102400 kbytes in   0.797209 sec =   128448 kbytes/sec
        middle:        102400 kbytes in   0.820908 sec =   124740 kbytes/sec
        inside:        102400 kbytes in   1.487073 sec =    68860 kbytes/sec

$ sudo diskinfo -vt /dev/da1
/dev/da1
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0988655    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.121221 sec =   32.485 msec
        Half stroke:      250 iter in   5.774391 sec =   23.098 msec
        Quarter stroke:   500 iter in   9.527913 sec =   19.056 msec
        Short forward:    400 iter in   2.549826 sec =    6.375 msec
        Short backward:   400 iter in   2.405381 sec =    6.013 msec
        Seq outer:       2048 iter in   0.203302 sec =    0.099 msec
        Seq inner:       2048 iter in   0.155567 sec =    0.076 msec
Transfer rates:
        outside:       102400 kbytes in   0.769086 sec =   133145 kbytes/sec
        middle:        102400 kbytes in   0.871308 sec =   117524 kbytes/sec
        inside:        102400 kbytes in   1.414511 sec =    72393 kbytes/sec

$ sudo diskinfo -vt /dev/da2
/dev/da2
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1013896    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
        Half stroke:      250 iter in   5.857010 sec =   23.428 msec

Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
        Half stroke:      250 iter in   5.857010 sec =   23.428 msec
        Quarter stroke:   500 iter in   9.475112 sec =   18.950 msec
        Short forward:    400 iter in   3.251636 sec =    8.129 msec
        Short backward:   400 iter in   2.026576 sec =    5.066 msec
        Seq outer:       2048 iter in   0.196214 sec =    0.096 msec
        Seq inner:       2048 iter in   0.151343 sec =    0.074 msec
Transfer rates:
        outside:       102400 kbytes in   0.692566 sec =   147856 kbytes/sec
        middle:        102400 kbytes in   0.893075 sec =   114660 kbytes/sec
        inside:        102400 kbytes in   1.587866 sec =    64489 kbytes/sec

$ sudo diskinfo -vt /dev/da3
/dev/da3
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1025747    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.027712 sec =   32.111 msec
        Half stroke:      250 iter in   5.888577 sec =   23.554 msec
        Quarter stroke:   500 iter in   9.404656 sec =   18.809 msec
        Short forward:    400 iter in   3.107148 sec =    7.768 msec
        Short backward:   400 iter in   3.314666 sec =    8.287 msec
        Seq outer:       2048 iter in   0.179241 sec =    0.088 msec
        Seq inner:       2048 iter in   0.154015 sec =    0.075 msec
Transfer rates:
        outside:       102400 kbytes in   0.802362 sec =   127623 kbytes/sec
        middle:        102400 kbytes in   0.843734 sec =   121365 kbytes/sec
        inside:        102400 kbytes in   1.547642 sec =    66165 kbytes/sec

$ sudo diskinfo -vt /dev/da4
/dev/da4
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1096918    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.036516 sec =   32.146 msec
        Half stroke:      250 iter in   5.829626 sec =   23.319 msec
        Quarter stroke:   500 iter in   9.222203 sec =   18.444 msec
        Short forward:    400 iter in   3.612478 sec =    9.031 msec
        Short backward:   400 iter in   2.545658 sec =    6.364 msec
        Seq outer:       2048 iter in   0.199791 sec =    0.098 msec
        Seq inner:       2048 iter in   0.147488 sec =    0.072 msec
Transfer rates:
        outside:       102400 kbytes in   0.715039 sec =   143209 kbytes/sec
        middle:        102400 kbytes in   0.816365 sec =   125434 kbytes/sec
        inside:        102400 kbytes in   1.467947 sec =    69757 kbytes/sec
```


----------



## Terry_Kennedy (Jan 20, 2013)

cbunn said:
			
		

> Well, it would seem that all five of my drives have exactly the same model suffix:


Agreed. What does smartctl show for the firmware version? You have a slightly larger spread of serial numbers than I'd normally expect for a bunch of drives ordered at the same time. If you have the drives out of the enclosure for any reason, you might want to also check the manufacture dates. I get 11/29/2012 for 4 of them and 12/05/2012 for one of them, but that may not be entirely accurate (if you registered them on the WD site, that's the date of registration, not the date of manufacture).



> I ordered my drives from Amazon and they were properly packed. Each drive was protected by the molded plastic end caps within individual cardboard boxes, lined up inside a larger box with airbags to pad any extra space.


That's standard retail packaging, which is what I'd expect from Amazon. The problem happens when some other vendors buy 20-pack cartons (which look like this) and then resell the individual drives. In that case, there's no unit piece factory packaging.



> HD Tune is a Windows program only, right? I figured diskinfo and bonnie++ would have been the equivalent of that.


Yes, it is Windows only (unfortunately). The advantage it has is that it times how long it takes to read each sector, and can thus provide a nice graph which will indicate any "slow" spots on the drive.


----------



## cbunn (Jan 20, 2013)

Terry_Kennedy said:
			
		

> What does smartctl show for the firmware version?




```
$ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Firmware Version"; done
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
```



			
				Terry_Kennedy said:
			
		

> You have a slightly larger spread of serial numbers than I'd normally expect for a bunch of drives ordered at the same time. If you have the drives out of the enclosure for any reason, you might want to also check the manufacture dates. I get 11/29/2012 for 4 of them and 12/05/2012 for one of them, but that may not be entirely accurate (if you registered them on the WD site, that's the date of registration, not the date of manufacture).


I'll give a look next time I'm in there. How did you get those dates, though? Some kind of code hidden in the serial numbers?




			
				Terry_Kennedy said:
			
		

> That's standard retail packaging, which is what I'd expect from Amazon. The problem happens when some other vendors buy 20-pack cartons (which look like this) and then resell the individual drives. In that case, there's no unit piece factory packaging.


I've definitely been down that road. I love the Egg, but they really drop the ball when it comes to packing some things.




			
				Terry_Kennedy said:
			
		

> Yes, it is Windows only (unfortunately). The advantage it has is that it times how long it takes to read each sector, and can thus provide a nice graph which will indicate any "slow" spots on the drive.


Nifty. But I don't see any way to pull that off with my set-up, I'm sad to say.


----------



## Terry_Kennedy (Jan 20, 2013)

cbunn said:
			
		

> ```
> $ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Firmware Version"; done
> Firmware Version: 80.00A80
> Firmware Version: 80.00A80
> ...


Nope, just looked up the warranty end dates on wdc.com and subtracted 3 years.


----------



## cbunn (Jan 20, 2013)

Terry_Kennedy said:
			
		

> Well, they're all the same - I don't have any idea why they would perform differently, unless some of them were behind a port expander (which could be on the motherboard).


They're all on a Supermicro AOC-USAS2-L8i card. No port expanders.



			
				Terry_Kennedy said:
			
		

> Nope, just looked up the warranty end dates on wdc.com and subtracted 3 years.


Oh. Well know I that one can do that.


----------



## cbunn (Jan 21, 2013)

So I've setup the RAIDZ zpool on the five drives using the instructions found on these forums and elsewhere using the temporary gnop definitions to force ZFS into using 4k sectors. To get an idea of speeds, I transferred the 9.1-Release memstick image (733MB) from the gmirrored USB flash drives to the new zpool within the time command.

```
$ time cp FreeBSD-9.1-RELEASE-amd64-memstick.img /storage/tmp/

real    0m0.585s
user    0m0.000s
sys     0m0.467s
```

But that implies that the transfer was at 1.2GB/s. That can't be right. So I did a few runs of bonnie++ to get a better idea of performance.






This shows an average write speed of 352MB/s and an average read of 326MB/s. Over the three instances, the writes were very close together, but the three reads had a lot of variation. I'm not sure what that's about.


----------



## wblock@ (Jan 21, 2013)

Looks good to me.  My three-drive RAIDZ does something like 200-220M/second reads.


----------



## Rubel (Jan 22, 2013)

Motherboard seems to be out of stock in the couple of places I've looked. Is there another ITX C20* server option you folks know of?


----------



## cbunn (Jan 22, 2013)

Rubel said:
			
		

> Motherboard seems to be out of stock in the couple of places I've looked. Is there another ITX C20* server option you folks know of?



Well, there is an update to the S1200KP named the S1200KPR. The main difference is that the KPR board supports Ivy Bridge CPUs out of the box (there are reports that upgrading the BIOS on the KP can allow the same). Provantage has it for $142.33, which is a pretty good price. I'm not sure what they charge for shipping.

Apart from those two boards, I've not seen any others that support an Intel CPU with ECC memory. And I searched quite a bit.


----------



## Rubel (Jan 30, 2013)

Thanks for the info! After some further reading, I've ordered a S1200KRP board and three WD Red drives for my own smaller version of what you've built. I wonder if I can skip the SAS controller and just use the on-board SATA ports plus a Rocker 620A 2 port card I have laying around. Although I don't see that card listed in the hardware compatibility section of the handbook, but google does list it working with some people's installations. If it develops problems and I have to move to another controller, I can just zfs export then zfs import my pool, right?


----------



## cbunn (Jan 30, 2013)

Rubel said:
			
		

> Thanks for the info! After some further reading, I've ordered a S1200KRP board and three WD Red drives for my own smaller version of what you've built. I wonder if I can skip the SAS controller and just use the on-board SATA ports plus a Rocker 620A 2 port card I have laying around. Although I don't see that card listed in the hardware compatibility section of the handbook, but google does list it working with some people's installations. If it develops problems and I have to move to another controller, I can just zfs export then zfs import my pool, right?



The motherboard has four SATA ports. Other than the three Red drives, what else will you be attaching?

I don't know a whole lot about that controller card, so I can't comment on its compatibility. But if you do get it set up and then you decide to replace it, a ZFS export/import should work fine.


----------



## Rubel (Jan 31, 2013)

cbunn said:
			
		

> The motherboard has four SATA ports. Other than the three Red drives, what else will you be attaching?


Oh, I just wanted everything to be on SATA III--two of the ports are SATA II, right? But wait, didn't I read somewhere that only SSDs even run fast enough to keep up with SATA III? Maybe I can forget the card altogether and just leave a SATA III port free for a potential future ZIL SSD.

Anyhow, once it's put together I can post some bonnie++ tests to compare to yours if that's interesting to anyone.


----------



## wblock@ (Jan 31, 2013)

Rubel said:
			
		

> Oh, I just wanted everything to be on SATA III--two of the ports are SATA II, right? But wait, didn't I read somewhere that only SSDs even run fast enough to keep up with SATA III? Maybe I can forget the card altogether and just leave a SATA III port free for a potential future ZIL SSD.



SATA II is good for a theoretical 300M/second.  That's far above anything a standard hard drive does.



> Anyhow, once it's put together I can post some bonnie++ tests to compare to yours if that's interesting to anyone.



Please do.


----------



## Rubel (Feb 1, 2013)

Oops, looks like I ordered registered RAM instead of unbuffered. So much for getting it running this weekend.


----------



## c_geier (Feb 1, 2013)

cbunn said:
			
		

> Well, there is an update to the S1200KP named the S1200KPR. The main difference is that the KPR board supports Ivy Bridge CPUs out of the box (there are reports that upgrading the BIOS on the KP can allow the same). Provantage has it for $142.33, which is a pretty good price. I'm not sure what they charge for shipping.
> 
> Apart from those two boards, I've not seen any others that support an Intel CPU with ECC memory. And I searched quite a bit.



Supermicro boards like the X9SAE support current Intel (non Xeon) CPUs and ECC Ram. While they are a bit more expensive, they have 4 DIMM sockets, 6 SATA ports and more PCIe and PCI slots.


----------



## c_geier (Feb 1, 2013)

@cbunn: Do you have a power meter and can measure idle power usage? I would be really interested in that.


----------



## cbunn (Feb 2, 2013)

c_geier said:
			
		

> Supermicro boards like the X9SAE support current Intel (non Xeon) CPUs and ECC Ram. While they are a bit more expensive, they have 4 DIMM sockets, 6 SATA ports and more PCIe and PCI slots.



Rubel was asking about Mini-ITX boards, which narrows the selection down considerably.



			
				c_geier said:
			
		

> @cbunn: Do you have a power meter and can measure idle power usage? I would be really interested in that.



Using my trusty Kill-A-Watt, I get about 50W at the outlet when idle. When fully loaded, it gets close to 70W. Not bad, I think.


----------

