# Recommended SATA expanders



## Gauss013 (Feb 16, 2015)

Hello,

I am new to FreeBSD, so my apologies if this is well documented somewhere. I have poked around here and can't seem to find a good database.

I have:

AMD FX-6300
ASUS M5A97-R2
5x 3.0 TB WD RED
1x 1.0 TB WD RED
1x Kingston SSD
24 GB Corsair Vengeance DDR3 

Seasonic 550W Power Supply
The M5A97-R2 only has 6 SATA ports, and ultimately I am configuring this beast to be a home server. I plan on tossing the 3.0 TB drives and the SSD into a ZFS pool, with the 1.0 TB RED being the OS drive.

What I need is an SATA-port expander, I don't really care about it's RAID support (ZFS makes that pointless)... I just need it to work.

I found this one: 
http://www.newegg.com/Product/Product.aspx?Item=N82E16816124064

I do not know anything about the Marvell 88SE9215 chipset's reliability within FreeBSD.

I have found recommendations on this forum about the SIL 34*/35* chipset, but many of those are multiple years old now. 

Any help is very much appreciated.


----------



## Oko (Feb 16, 2015)

A typical ZFS recommendation is not SATA expender but HBA (host bus adapter). Community favorite is LSI 9211-8i (new on E-bay goes for about $100). You can also buy IBM ServeRAID M1015 and flush to IT mode effectively making it LSI 9211-8i. That could save you $10-$20.

For RAID-Z2 you need 6 HDD for RAID-Z3 you will need 7HDD. You could install OS on 1TB and use SSD drive for ZIL and L2ARC? This is a good article http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flash-memory-ssds-and-zfs. Actually if you ask me I would add another 1TB HDD and have OS installed on RAID-Z mirror. Now that RAM is ECC? If not good luck running ZFS.


----------



## diizzy (Feb 16, 2015)

A SATA expander is something else than a controller, if you want something cheap the ASMedia 1061 controllers are actually quite decent from what I can tell and the seem better overall than Marvell ones. Downside is that it only does 2 drives per card. You can find these on Amazon and Newegg reasonably priced.
//Danne


----------



## wblock@ (Feb 16, 2015)

I have seen several reports of problems with port expanders.  Controllers with more ports are likely to be better tested.  The LSI controllers are well-regarded.  I've used the cheap SIL3124 four-port cards with success.  They are not new, and definitely not a high-end controller, but they do work.

The SSD will probably not make a very noticeable difference in a home server.


----------



## Gauss013 (Feb 16, 2015)

Oko said:


> A typical ZFS recommendation is not SATA expender but HBA (host bus adapter). Community favorite is LSI 9211-8i (new on E-bay goes for about $100). You can also buy IBM ServeRAID M1015 and flush to IT mode effectively making it LSI 9211-8i. That could save you $10-$20.



Thanks, I've heard about the IBM ServeRAID, but not the flush to IT mode. I'll probably just pick up an LSI 9211-8i, it was recommended by a few others too.



Oko said:


> For RAID-Z2 you need 6 HDD for RAID-Z3 you will need 7HDD. You could install OS on 1TB and use SSD drive for ZIL and L2ARC? This is a good article http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flash-memory-ssds-and-zfs. Actually if you ask me I would add another 1TB HDD and have OS installed on RAID-Z mirror. Now that RAM is ECC? If not good luck running ZFS.



My intention was to run a standard RAID-Z. I don't want to get into anything with double-redundancy... I understand the rationale, but for my personal usage, I find anything more than double-fault tolerance to be excessive for the money. Actually when I crunched the numbers, the most cost effective thing for me (since I don't have a high availability requirement) is to have 2 sets of 5-drive RAID-Zs, with one being a "hard-backup." 

Intention was the SSD is going to act as a ZIL. 

The RAM is not ECC... Was trying to re-use some RAM from my gaming rig to avoid spending stupid amounts of money. Looks like that was a bad idea... I hadn't considered the ECC element.


----------



## diizzy (Feb 16, 2015)

ZFS will run fine without ECC but not just as reliable as it could be...
//Danne


----------



## wblock@ (Feb 16, 2015)

Gauss013 said:


> but not the flush to IT mode



That was probably meant to be _flash_.

Many people use ZFS with non-ECC RAM, or only 4G.  Some experience failures, but it does not seem to be a very high proportion.  A bigger concern could be insufficient redundancy in the drives.  The magic number is supposed to be 12TB.  At that size, the failure rate of drives is supposed to overtake single redundancy.  One drive fails, another is swapped in, and before the new drive is finished resilvering, another drive fails.  Most manufacturer ratings have little relationship with reality, so how safe a five-drive RAIDZ1 will really be is unknown.


----------



## Gauss013 (Feb 17, 2015)

wblock@ said:


> That was probably meant to be _flash_.
> 
> Many people use ZFS with non-ECC RAM, or only 4G.  Some experience failures, but it does not seem to be a very high proportion.  A bigger concern could be insufficient redundancy in the drives.  The magic number is supposed to be 12TB.  At that size, the failure rate of drives is supposed to overtake single redundancy.  One drive fails, another is swapped in, and before the new drive is finished resilvering, another drive fails.  Most manufacturer ratings have little relationship with reality, so how safe a five-drive RAIDZ1 will really be is unknown.



Do you have any white-papers or links on this? I'd like to do more research, I've read this argument before... But it seems to never make sense in my brain. How does a double-redundant RAID help this specific problem? You still have the failure, you still resilver, and during the resilvering, another drive still fails (and thus keep the cycle going). I am worried about this setup, particularly because I am at that magic number (my setup in a Z1 -> 4x 3.0TB of data, 1x 3.0 TB of parity). I'm trying to do the best I can without spending 1200 bucks on server grade parts (I bought WD Red drives in an effort to stem some of this problem).  

Granted, I am not an expert on file-systems, but (from my perspective) the resilvering process is largely the same for Z2/Z3... You compare the RAID array with the parity bytes to determine what the new data should be, etc, etc. From my perspective the advantage of Z2/Z3 always seemed to be the fault tolerance, Z2 requires 3 faults before (theoretically) unrecoverable data loss and Z3 requires 4 faults before (theoretically) unrecoverable data loss.

I suppose if I ignore the ECC aspect, a 6th drive is only another 100 bucks... But then the question becomes would I be better off with a 6th drive or ECC RAM? Or am I spending money chasing a 99.9999% reliability vs. 99.999999% reliability.


----------



## Oko (Feb 17, 2015)

Gauss013 said:


> I suppose if I ignore the ECC aspect, a 6th drive is only another 100 bucks...


I know that people are going to flame me for this comment but if the money is of any concern (don't get me wrong even business are very careful how they spend money but they will not try to save $1000 if that is going to cause troubles) I would very strongly suggest that you look HAMMER which currently runs only on DragonFlyBSD.


----------



## wblock@ (Feb 17, 2015)

Gauss013 said:


> Do you have any white-papers or links on this?


The original article: http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/.
Another take: http://www.high-rely.com/blog/why-raid-5-stops-working-in-2009-not/.

It is about the amount of space versus the amount of redundancy and the failure rate.  In a typical three-drive RAID, when one drive fails, the data is still safe.  But now there is no redundancy.  Resilvering is going to take a while.  If there is a bit error in one of the remaining two original drives before resilvering finishes, data is lost.  With the manufacturer-reported error rates, and the sizes of drives now, 12TB is where it starts to become likely to have another error before resilvering finishes.

Now, ZFS is not like standard RAID, so that can change the numbers.  The second article above says that reality is different from the manufacturer-predicted bit error rates, so that can change it also.  The question becomes what the data is worth.


----------



## wblock@ (Feb 17, 2015)

Oko said:


> I know that people are going to flame me for this comment but if the money is of any concern (don't get me wrong even business are very careful how they spend money but they will not try to save $1000 if that is going to cause troubles) I would very strongly suggest that you look HAMMER which currently runs only on DragonFlyBSD.



Hammer on Dragonfly has a very small user and developer base and can't be reasonably compared to ZFS on FreeBSD at present.  Please suggest this again five or six years after Hammer 2 has been released and is in wide use.


----------



## Gauss013 (Feb 17, 2015)

Oko said:


> I know that people are going to flame me for this comment but if the money is of any concern (don't get me wrong even business are very careful how they spend money but they will not try to save $1000 if that is going to cause troubles) I would very strongly suggest that you look HAMMER which currently runs only on DragonFlyBSD.



I know, trust me, if I was doing this for something other than my own personal server... I'd be willing to spend more. Unfortunately, as with many home projects, I have to function within a budget. I am currently migrating a media server that was built on a debian distro I ran on a Wandboard with a dedicated RAID5 enclosure to this for various reasons (speed and space being the big ones, but the idea was spawned by the zdnet article wblock linked).



wblock@ said:


> The original article: http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/.
> Another take: http://www.high-rely.com/blog/why-raid-5-stops-working-in-2009-not/.
> 
> It is about the amount of space versus the amount of redundancy and the failure rate.  In a typical three-drive RAID, when one drive fails, the data is still safe.  But now there is no redundancy.  Resilvering is going to take a while.  If there is a bit error in one of the remaining two original drives before resilvering finishes, data is lost.  With the manufacturer-reported error rates, and the sizes of drives now, 12TB is where it starts to become likely to have another error before resilvering finishes.
> ...



I actually read the first one before, it is what got me started thinking about this whole topic. Ultimately I didn't pull the trigger on this pet-project for almost a year-and-a-half, but I was researching what to do about my media server. My embedded devices were becoming increasingly less likely to satisfy my requirements, despite being significantly more power-efficient (My old media server, based on my steady-state power calculations, requires only ~20% of this FreeBSD build I am making).

I guess I never connected the dots because of your last point... ZFS is kind of a different animal, it is a file-system that seems to "coexist" with the RAID concept. Atleast based on my outside knowledge. I get where you are going now though, I guess I hadn't realized it before; the issue is the error probability on the drives when NOT protected by parity (which would be the case during resilvering). Adding a second layer of fault tolerance would (theoretically) reduce the likelihood of data loss by an order of magnitude. Makes sense, guess I got a lot of reading to do to figure out where to take this beast.


----------



## Oko (Feb 17, 2015)

wblock@ said:


> Hammer on Dragonfly has a very small user and developer base and can't be reasonably compared to ZFS on FreeBSD at present.


And I am not comparing it to ZFS. The guy is going to run his private file server. I am not suggesting by any stretch of imagination that you should start moving the servers in your data center from FreeBSD to DragonFly.  Even I in a relatively small shop have had troubles using DragonFly due to the missing enterprise infrastructure. However you are wrong about stability of HAMMER 1. It is rock stable and tested by many people in production for over 5-6 years although definitely not as much as ZFS.


----------



## ondra_knezour (Feb 17, 2015)

Some reading about ECC and memory errors probability 
http://lambda-diode.com/opinion/ecc-memory
http://www.zdnet.com/article/dram-error-rates-nightmare-on-dimm-street/
http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf


----------



## usdmatt (Feb 17, 2015)

The main issue with HAMMER would be that it doesn't do any volume management so you'd have to use a RAID controller. Ok, but the downside with FreeBSD is always how well supported it is, and whether you can actually get any monitoring tools or software to work properly. The beauty of ZFS is that it's well supported and all software based, so everything is easily managed from the FreeBSD command line. If necessary you can move the whole array to another machine without worrying about any RAID hardware and just pick the pool up again.

Yes, ECC is great is you want to guarantee data integrity but a hell of a lot of people are running ZFS without it, especially at home, including me. Maybe I have a few incorrect bits in some files somewhere but I don't think it's going to be the end of the world. All file systems are writing data that originates from RAM so just suggesting a different file system isn't exactly an instant fix. What would be interesting is actual research on whether ZFS is more susceptible to errors from non-ECC RAM when compared to other file systems. Statements like "Good luck running ZFS on non-ECC RAM" as if the ZFS pool is going to fall apart, then suggesting other file systems as if ECC is suddenly not an issue are a bit futile.

Expanders seem like a great idea and are fairly common in enterprise storage, but in enterprise it's usually all one manufacturer and been heavily tested. Unfortunately it seems that a lot of these devices have weird quirks which are probably handled fine by the tested controllers but we see a lot of people with strange problems on the forums. Because the set up of controller -> expander -> disks adds a lot of variables, many combinations of which have had little use or testing among community members, it's very difficult to support. On the forums you tend to just see stuff like "try a firmware upgrade" which is basically just clutching at straws in the hope you've hit a bug that's been fixed.

I would just go for a well supported HBA and as mentioned use RAID-Z2 if possible, if only for peace of mind.


> You still have the failure, you still resilver, and during the resilvering, another drive still fails (and thus keep the cycle going).


It's not so much the ability to lose two disks that's important, it's that time when you have had a single disk failure. If you lose a disk in RAID-Z1, you need to get a replacement. It may be that you have a spare, or you may need to order it. Once you have the disk you then need to do a resilver, which could take a couple of days with that size of pool. So your total time in degraded state could be several days. If you get any errors on a disk during that time (doesn't have to be a full failure, could just be a checksum error or bad block), the data that was being read is lost. ZFS will start showing "errors detected. applications are affected" warnings and will start building a list of files that are corrupt as it has no redundancy to recreate them from. Not the end of the world but not very nice either. The possibility of errors happening during those degraded days on arrays ~>10 TB is high enough that in general, RAID5/RAID-Z1 is discouraged. With RAID-Z2, if you get any read or checksum errors while that one disk is missing or rebuilding, ZFS will just recover the data automatically as normal.

I'm not saying you outright can't use RAID-Z1 if it makes sense for your budget and number of disks, just be aware that it puts you in a more dangerous position when you are replacing a failed disk.

*Obviously you'll still want to backup any data that can't be recreated easily.*
As with all RAID, redundancy is not a backup.


----------



## Oko (Feb 17, 2015)

usdmatt said:


> The main issue with HAMMER would be that it doesn't do any volume management so you'd have to use a RAID controller. Ok, but the downside with FreeBSD is always how well supported it is, and whether you can actually get any monitoring tools or software to work properly.


You pretty much hit the nail in the head. To be even more specific it is not even difficult to find RAID controller which works well on DF (Areca is the best tested by DF community) but monitoring DF and in particular RAID controller is hit and miss. ZFS also has advantage of supporting compression (DF has GSoC to add such support to HAMMER-1) and the fact that snapshots are writeable while hammer mirror streams are not. However HAMMER history hands down beats the pants out of ZFS journalling.


----------



## diizzy (Feb 17, 2015)

wblock@

SATA/SAS expanders can work very good but you need to be careful how you mix and match controllers and expanders. Apparently HPs plays nice with LSI cards in general. http://hardforum.com/showthread.php?t=1484614
//Danne


----------



## wblock@ (Feb 17, 2015)

diizzy, my concern with port expanders is mostly specific to FreeBSD.  There might be lots of people using them without complaints, but the only time I've seen them mentioned is when people say they had problems with port expanders.


----------



## Terry_Kennedy (Feb 18, 2015)

ondra_knezour said:


> Some reading about ECC and memory errors probability
> http://lambda-diode.com/opinion/ecc-memory
> http://www.zdnet.com/article/dram-error-rates-nightmare-on-dimm-street/
> http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf


I have about 2 dozen servers, all with ECC memory. Memory per box varies from 4GB to 96GB. In the last 5 years, none have ever recorded a correctable (or uncorrectable) memory error. I don't have detailed records going back further than that, but I do remember *one* instance of a correctable error on a Tyan S2721-533 (a Socket 603 board) about 12 years ago.

That's just anecdotal information, of course. These are all server-class hardware and almost all of them require ECC memory and won't work without it.


----------



## User23 (Feb 18, 2015)

I am just wondering about the use of the word "SATA Expanders". I see no expanders here, I only see SATA port multipliers and HBAs.
Personally I thought expanders means SAS/SATA Backplanes which can fully access each drive. Please correct me if I am wrong, I have never seen a SATA only expander. There are only SAS Expanders that are able to talk SATA drives through SAS, but need a SAS HBA.


----------



## ondra_knezour (Feb 18, 2015)

User23 you are of course right, to expand and to multiply are used here as alternatives, because result is the same from uniformed point of view - multiple devices connected to only one port. There is nice review of options on http://www.sasexpanders.com/vs/ for those who want to compare them.


----------



## wblock@ (Feb 18, 2015)

Sorry, I was really talking about port multipliers.


----------



## diizzy (Feb 18, 2015)

http://ascii.jp/elem/000/000/962/962694/
http://www.kuroutoshikou.com/product/interface/ata_sata/sata3i10-pcie/release/
Looks really interesting and not very expensive, ~8000 Yen (slightly less than 80 USD)

Russian:
http://www.overclockers.ru/hardnews...ci-e-2-0-x2-obespechivajuschaya-10-sata3.html
Translated:
https://translate.google.com/translate?sl=ru&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http://www.overclockers.ru/hardnews/65998/pm-pcie10sata3-plata-rasshireniya-pci-e-2-0-x2-obespechivajuschaya-10-sata3.html&edit-text=&act=url
//Danne


----------



## ondra_knezour (Feb 18, 2015)

(I would assume that most of visitors here have little to none knowledge about yen exchange rates and even fewer are proficient in japan reading)


----------



## User23 (Feb 19, 2015)

2 Lanes of PCIe 2.0 = 1 GByte/s 
10x SATA3 = 10 x 6Gbit/s = 7,5 GByte/s

So even if the chips are supported, you don't want to connect more than 1 SSDs if you want to use the rest of the SATA Ports for slow HDDs.


----------



## mav@ (Feb 19, 2015)

diizzy said:


> http://ascii.jp/elem/000/000/962/962694/
> http://www.kuroutoshikou.com/product/interface/ata_sata/sata3i10-pcie/release/
> Looks really interesting and not very expensive, ~8000 Yen (slightly less than 80 USD)
> 
> ...



That card looks like combination of AHCI HBA, and two SATA port multipliers. So in theory that it should work on FreeBSD, since all required functionality probably in place. But that is only in theory. In practice SATA port multipliers significantly complicate I/O operation and error recovery. So I would not bet on reliability of that combination. Performance is also questionable there -- even not looking on PCIe x2 connector, simultaneous access to disks connected to the same port multiplier requires advanced functionality from HBA, and I am not sure that chip used there supports it.

I would probably not put such device into critical production, but I would be quite interested to read any review of such device under FreeBSD. I could even buy it just for fun if it would be sold in nearby shop.


----------



## diizzy (Mar 11, 2015)

Looks like it's finally available!

http://www.addonics.com/products/ad10sa6gpx2.php
http://www.shopaddonics.com/Itemdesc.asp?ic=AD10SA6GPX2&eq=&Tp=
http://www.aimtec.co.uk/addonics-on...llers/6g-10-port-sata-pcie-2x-controller.html

//Danne


----------

