# Building up a 16TB system



## dvl@ (Nov 5, 2012)

Hello folks,

I'm looking at putting together a new box, mostly for storage.  But it will probably run some virtual machines, and be used for some regression testing of backup software and for developing various website.  The box will be sitting in my basement, not in a data center.

I've been looking at the components, I'm looking to order within the next week or two.  The system will be running FreeBSD 9.  The main OS will be installed on two SSDs and the main storage will be spread across 8x2T HDDs clustered together into a raidz2 ZFS array.

Here are the components I'm considering.  The links below are all Newegg, and I like dealing with them.

* SUPERMICRO MBD-H8SGL-O ATX Server Motherboard : $224.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182240

* AMD Opteron 6128 Magny-Cours 2.0GHz 8 x 512KB L2 Cache 12MB L3 Cache Socket G34 115W 8-Core Server : $259.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16819105266

* Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1600 Server Memory : 4 x $59.99 = $239.96
http://www.newegg.com/Product/Product.aspx?Item=N82E16820239230

* PC Power and Cooling Silencer MK III 600W power supply : $99.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16817703036

* Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB : 8 x $114.99 = $919.92
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834

* Mushkin Enhanced Chronos MKNSSDCR60GB 2.5â€³ 60GB SATA III Asynchronous MLC Internal Solid State Drive (SSD) : $59.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16820226247

* Corsair Force Series GT CSSD-F60GBGT-BK 2.5â€³ 60GB SATA III Internal Solid State Drive (SSD) :$89.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16820233193

* LIAN LI PC-A70F USB3.0 Black Aluminum ATX Full Tower Computer Case : $189.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16811112342



Not listed above is the disk controller.  The controller will be used in pass-through mode (i.e. JBOD), not RAID.  I've considered these cards:


* SYBA SD-PEX40054 PCI-Express 2.0 x4 SATA III (6.0Gb/s) HyperDuo 4-port RAID Controller Card $41.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16816124056

* SUPERMICRO AOC-USAS2-L8i PCI Express SATA / SAS Eight-Port Internal RAID Adapter
http://www.newegg.com/Product/Product.aspx?Item=N82E16816101334

* HighPoint Rocket 640L PCI-Express 2.0 x4 Low Profile SATA III (6.0Gb/s) RAID Controller Card : 2 x $69.99= $149.98
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115114

* Intel RAID Controller Card 6G SAS PCI-E x8 8 internal ports (RS2WC080)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117171&Tpk=rs2wc080 $257.99


Any comments on the selection?  Thanks.


----------



## Sfynx (Nov 5, 2012)

I think it would be more cost effective to get a Supermicro board with the 8-port 6 Gbps LSI SAS built right into the board. Then you have the board and disk controllers for a couple hundred bucks. E.g. an X8SI6-F (or -O if you don't care about IPMI), or some AMD equivalent.


----------



## dvl@ (Nov 5, 2012)

That's a good idea.

But even with 8 ports onboard, I still need two more ports for the gmirror I plan for the base OS.

I did find this 'gamer' m/b: ASUS Crosshair V (http://www.newegg.com/Product/Product.aspx?Item=N82E16813131876) for $230

I also tried searching for AMD boards by Supermicro, with more SATA III, but failed.

Key here is support for ECC RAM.  I want that.  It is important.


----------



## Sfynx (Nov 5, 2012)

The Supermicro boards I've seen normally have the SAS controller on top of a standaard set of 6 SATA ports, so that makes 14 ports  Haven't found a single CPU AMD board with this though, I have a Xeon board myself.
All server boards I came across have support for ECC RAM, some pro workstation boards support it also, regular consumer stuff generally doesn't. And yeah I agree, ECC RAM is mandatory for ZFS based servers imo, an unlucky bitflip corrupting a fsck-less storage system is not cool


----------



## _martin (Nov 5, 2012)

Hi, 
Recently I bought the same HDD you are aiming for: ST2000DM001. I had big problems with it (probably servo), weird noises coming out from time to time causing little response delay. I did put it through RMA process 3x times. Each time I got new disk. Each time the same issues. All of them had firmware version CC4C.

I never had problems with Seagate disks before, but this model drove me crazy.


----------



## dvl@ (Nov 5, 2012)

matoatlantis said:
			
		

> Hi,
> Recently I bought the same HDD you are aiming for: ST2000DM001. ..... All of them had firmware version CC4C.



How recent?

Did you try upgrading the firmware?  That was the first thing to do, based on a recommendation elsewhere.


----------



## dvl@ (Nov 5, 2012)

Sfynx said:
			
		

> The Supermicro boards I've seen normally have the SAS controller on top of a standaard set of 6 SATA ports, so that makes 14 ports  Haven't found a single CPU AMD board with this though, I have a Xeon board myself.



I'm keen on AMD, but would go Intel if required....


----------



## wblock@ (Nov 5, 2012)

I would prefer the Seasonic power supplies to anything else.  The M12II 620 is $10 less, the X650 is $20 more.

Mixing brands on the SSDs is a good idea, but you may also want to mix internal controller types.  Those are both Sandforce.  Plextor, Crucial, some Intel drives, and Samsung use non-Sandforce controllers.  I've had good results with Plextor and Intel.

And then the hard drives... other than WD Black, I just don't know.  Seagate has given me a bad feeling lately, and their drives usually only have a one-year warranty.  The WD Red drives are well-regarded and made for RAID, but are too new to have much of a track record.  They are overpriced at Newegg.  If possible, mixing brands on the 2T drives could be a good idea also.


----------



## _martin (Nov 5, 2012)

dvl@ said:
			
		

> How recent?
> 
> Did you try upgrading the firmware?  That was the first thing to do, based on a recommendation elsewhere.



For the first one, yes. Sorry I can't give you more detailed info, I don't remember. I pushed it to the latest firmware available. The other two, I just tested and returned when the same issue occurred.  

Dates (all year 2012): 

```
1st: Mar 12
2nd: Apr 25
3rd: Jun 11
```


----------



## Sfynx (Nov 5, 2012)

btw, RAID-Z2 is not very strong on small random I/O especially when you're going to mix all kinds of different drives (good for preventing bad batch problems though), so if you expect such a workload it could be beneficial to go for a bit bigger SSD option so you can reserve some space for L2ARC (does not need to be mirrored).


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> Mixing brands on the SSDs is a good idea, but you may also want to mix internal controller types.  Those are both Sandforce.  Plextor, Crucial, some Intel drives, and Samsung use non-Sandforce controllers.  I've had good results with Plextor and Intel.



Good point. Interesting.  I hadn't considered that.


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> And then the hard drives... other than WD Black, I just don't know.  Seagate has given me a bad feeling lately, and their drives usually only have a one-year warranty.  The WD Red drives are well-regarded and made for RAID, but are too new to have much of a track record.  They are overpriced at Newegg.  If possible, mixing brands on the 2T drives could be a good idea also.



I recognize the theory behind mixing HDD brands, but with 8 drives, if I used 4 different brands, then two failures with raidz2... I'm still OK.  With 2 or three brands... no so much.


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> I would prefer the Seasonic power supplies to anything else.  The M12II 620 is $10 less, the X650 is $20 more.



Why do you like this brand?


----------



## dvl@ (Nov 5, 2012)

Based on 4 brands:

TOSHIBA DT01ACA200 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $110
http://www.newegg.com/Product/Product.aspx?Item=N82E16822149407

Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $115
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834

Western Digital WD Black WD2002FAEX 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $179
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136792

Hitachi GST Ultrastar 7K3000 HUA723020ALA640 (0F12455) 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s  2x $240
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145476


Total (without shipping/tax): $1288



Based on two brands (the first two listed above):

TOSHIBA DT01ACA200 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $110
http://www.newegg.com/Product/Product.aspx?Item=N82E16822149407

Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $115
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834


Total (without shipping/tax): $900

I'm not so sure how much having two brands will help. I think any risk avoidance is minimal.


----------



## gkontos (Nov 5, 2012)

For an HBA controller you might want to consider:

http://www.lsi.com/products/storagecomponents/Pages/LSISAS9207-8i.aspx
http://www.lsi.com/products/storagecomponents/Pages/LSISAS9211-8i.aspx

They both run fantastic under FreeBSD 9-STABLE and 9.1-RCX

For Hard Disks I have very good experience in performance with the WD RED. However, it is too soon to tell how reliable they are. So far I don't have any of them failing (46 total)


----------



## gkontos (Nov 5, 2012)

dvl@ said:
			
		

> Based on 4 brands:
> Western Digital WD Black WD2002FAEX 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $179
> http://www.newegg.com/Product/Product.aspx?Item=N82E16822136792



RMA 14 units this year! Performance really sucked.



			
				dvl@ said:
			
		

> Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $115
> http://www.newegg.com/Product/Produc...82E16822148834



No problems what so ever. Limited units to test though. (6)


----------



## wblock@ (Nov 5, 2012)

Seasonic is very well-regarded for build quality in hardware repair circles.  Most PC Power and Cooling supplies actually are (were?) based on Seasonic OEM.  Compare:

http://www.newegg.com/Product/Product.aspx?Item=N82E16817703036
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151095


----------



## wblock@ (Nov 5, 2012)

If I had to buy two brands of drives, I might get the Seagate "D-moo" drives and the WD Reds.  Toshiba I avoid, not that they've failed on me, just that every one I've ever had was so slow that I wished for it to fail.  Hitachi seems to think their questionable reliability is worth more than others.

For an array for size rather than speed, I might consider the WD Greens.  The newest ones with 64M cache are supposed to, well, suck less than the earlier ones.


----------



## dvl@ (Nov 5, 2012)

gkontos said:
			
		

> For an HBA controller you might want to consider:
> 
> http://www.lsi.com/products/storagecomponents/Pages/LSISAS9207-8i.aspx
> http://www.lsi.com/products/storagecomponents/Pages/LSISAS9211-8i.aspx
> ...



My reading shows that the 9207arrives in IT mode (meaning pass-through/JBOD mode), which is just what is needed for a ZFS array.

I can't tell yet about the 9211, but this post seems to indicate that it does.

NOTE: But this post indicates that you need to replace the 9211 firmware with an IT version

Any knowledge on these points?


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> If I had to buy two brands of drives, I might get the Seagate "D-moo" drives and the WD Reds.  Toshiba I avoid, not that they've failed on me, just that every one I've ever had was so slow that I wished for it to fail.  Hitachi seems to think their questionable reliability is worth more than others.
> 
> For an array for size rather than speed, I might consider the WD Greens.  The newest ones with 64M cache are supposed to, well, suck less than the earlier ones.




I cannot find D-moo drives...

WD Reds are about +50% over other similar drives...

I've had 8x2TB Hitachi drives in another system for about 2 years.  Fine so far.

re WD Green: Yeah, to be fair, I'm trying for speed, but I'm not sure I'll get there with this combination.  There are so many variables, that I'm considering going with the lower end components and just let it be...


----------



## gkontos (Nov 5, 2012)

dvl@ said:
			
		

> My reading shows that the 9207arrives in IT mode (meaning pass-through/JBOD mode), which is just what is needed for a ZFS array.
> 
> I can't tell yet about the 9211, but this post seems to indicate that it does.
> 
> ...



You can always flash the firmware. So far I have only done that on a LSI MegaRaid controller that was having issues with JBOD.


----------



## wblock@ (Nov 5, 2012)

dvl@ said:
			
		

> I cannot find D-moo drives...



Seagate STnnnn*DM00*1. 



> WD Reds are about +50% over other similar drives...



At Newegg, yes.  At other places, they're fair.



> re WD Green: Yeah, to be fair, I'm trying for speed, but I'm not sure I'll get there with this combination.  There are so many variables, that I'm considering going with the lower end components and just let it be...



It would be fun to build two arrays, one higher-end, and one the absolute cheapest, do benchmarks, and run them for a while.


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> It would be fun to build two arrays, one higher-end, and one the absolute cheapest, do benchmarks, and run them for a while.



I will accept donations to accept this challenge.


----------



## dvl@ (Nov 5, 2012)

wblock@ said:
			
		

> Seagate STnnnn*DM00*1.



OH!  Well, it seems I can get Enterprise drives for about +$5 each..

Compare on this Seagate Constellation ST1000NM0001 on Amazon versus my original choice ST2000DM001 on Newegg

NOTE: that first drive is SAS, the original is SATA.

That said, that same drive on Amazon is $90.

As you pointed out, I need to shop around.


----------



## dvl@ (Nov 5, 2012)

Please ignore the above post.  I am comparing a 1TB SAS with a 2TB SATA.  Sorry.


----------



## Sfynx (Nov 5, 2012)

dvl@ said:
			
		

> My reading shows that the 9207arrives in IT mode (meaning pass-through/JBOD mode), which is just what is needed for a ZFS array.
> 
> I can't tell yet about the 9211, but this post seems to indicate that it does.
> 
> ...



Flashing the firmware is easy enough, the integrated SAS controller on my Supermicro X8SI6-F board (which is essentially an LSI 9211-8i with spoofed PCI ID) came in IR mode and I could just reflash it using LSI's firmware.


----------



## dvl@ (Nov 5, 2012)

Sfynx said:
			
		

> Flashing the firmware is easy enough, the integrated SAS controller on my Supermicro X8SI6-F board (which is essentially an LSI 9211-8i with spoofed PCI ID) came in IR mode and I could just reflash it using LSI's firmware.



I'm beginning to like this board.  But given that I'm going to have 8x SATA III HDD drives and 2x SSD drives... 

Looking at the Newegg page indicates it comes with the cables for all 8 drives connected to the SAS controller.

I take it that's what you are using?  SATA drives on the SAS contoller in JBOD?


----------



## Sfynx (Nov 5, 2012)

dvl@ said:
			
		

> I'm beginning to like this board.  But given that I'm going to have 8x SATA III HDD drives and 2x SSD drives...
> 
> Looking at the Newegg page indicates it comes with the cables for all 8 drives connected to the SAS controller.
> 
> I take it that's what you are using?  SATA drives on the SAS contoller in JBOD?



Yeah, the thing is hooked up to a 12-port SAS/SATA backplane with three SFF-8087 connected rows of 4 ports.
Two of those go to the SAS controller using SFF-8087 cables, and one row is split among 4 SATA ports using a bracket cable (I figured that if that's gonna be a problem in the future we can always get a SAS expander later)

That leaves 2 SATA ports for boot/cache drives.

It supports a lot less RAM though, max 16 GB unbuffered / 32 GB registered


----------



## dvl@ (Nov 6, 2012)

wblock@ said:
			
		

> Mixing brands on the SSDs is a good idea, but you may also want to mix internal controller types.  Those are both Sandforce.  Plextor, Crucial, some Intel drives, and Samsung use non-Sandforce controllers.  I've had good results with Plextor and Intel.



I'm finding that the Sandforce controllers tend to do:

Sustained Sequential Read  - Up to 555MB/s
Sustained Sequential Write - Up to 555MB/s

While the Samsung (& others) do:

Max Sequential Read - Up to 520 MB/s
Max Sequential Write - Up to 160 MB/s

It seems only Sandforce has those higher write speeds.


----------



## wblock@ (Nov 6, 2012)

dvl@ said:
			
		

> I will accept donations to accept this challenge.



Ha, as will I!  The WD Greens or the Seagates appear to be the cheapest.  The WD Reds are about 20% more, but it can be a problem finding a place that has them in stock.


----------



## dvl@ (Nov 6, 2012)

wblock@ said:
			
		

> Ha, as will I!  The WD Greens or the Seagates appear to be the cheapest.  The WD Reds are about 20% more, but it can be a problem finding a place that has them in stock.



The Seagates support NCQ...


----------



## wblock@ (Nov 6, 2012)

dvl@ said:
			
		

> I'm finding that the Sandforce controllers tend to do:
> 
> Sustained Sequential Read  - Up to 555MB/s
> Sustained Sequential Write - Up to 555MB/s
> ...



Yes, but there's a catch.  The Sandforce controllers only get those write speeds on compressible data.  Other controllers don't use compression.  Overall, they're likely faster but not as much as the numbers would suggest.  The Sandforce controllers just did not seem fully ready.  Then Intel brought out SSDs with their own custom Sandforce firmware this year.  If I had to pick a Sandforce SSD, I'd go with those.


----------



## dvl@ (Nov 6, 2012)

Sfynx said:
			
		

> That leaves 2 SATA ports for boot/cache drives.
> 
> It supports a lot less RAM though, max 16 GB unbuffered / 32 GB registered



Oh.   Hmm, I guess I'd need registered.  I was planning on 32GB to start....


----------



## dvl@ (Nov 6, 2012)

wblock@ said:
			
		

> Ha, as will I!  The WD Greens or the Seagates appear to be the cheapest.  The WD Reds are about 20% more, but it can be a problem finding a place that has them in stock.



I am becoming more partial to the WD Reds now. 

And yes, even Amazon will sell only 5 at at time...


----------



## boris_net (Nov 9, 2012)

Just curious what kind of transfer rates would you expect from such a build both locally and over the network?


----------



## dvl@ (Nov 9, 2012)

I have absolutely no idea.  

But if someone funds me the money, I'll do both a low-end server and a higher end server and compare the differences.  I think that'd be a very interesting comparison.


----------

