# ECC or not? AMD with ECC vs intel without, homserver, consumer parts, zfs.



## naguz (Jan 28, 2013)

So, the title says it all, albeit in keywords.

I'm about to upgrade my server currently running in my old desktop - LGA775, E6600 etc. It needs a new PSU and new disks anyway, and a new motherboard, so it's in for a full upgrade.

For PSU I'm thinking about a OCZ MODXSTREAM PRO 500W - overkill, but silent, pretty cheap, and reliable. (AFAIK)
For HDDs I'm thinking 3  3TB WD Reds.

Then there is the question about RAM. It seems to me that when buying RED drives and running ZFS storage for a hopefully painless experience with decent reliability, it would be a bad idea to go with anything but ECC. But, it's not like my data will be residing in RAM most of the time -  It's usually sitting on disk, and when read, is transfered out of the system. I've been weighing for and against for a few days here, and can't rally decide. Unbuffered ECC @ 1333Mhz is about the same as normal ddr3 on 1600Mhz, so the price doesn't really come into it, and then it seems silly to go for non-Ecc... But then I can't go with any cheap 1155 mobo and a i3 IB, and must find AMD parts, which I don't know much about. I'm not even sure if I'm debating AMD vs Intel or ECC vs. Non-ECC with myself anymore! Could someone come and make a decision for me or something? 

I'd like low power usage in idle, but wouldn't mind having a bit of oomph if for example I should decide to transcode videos or whatnot. And I'll probably run GELI on the disks with ZFS on top. I might prefer AMD if for no other reason than not buying into the future monopoly of Intel, all specs and other things equal.

Problem is, I have no clue about AMD performance (or lower intel like i3), and I have only managed to confuse myself even more with my compulsive googling for ECC vs non ECC and AMD and intel performance in a home server scenario. I'd really appreciate any advice and stress relief.


----------



## wblock@ (Jan 29, 2013)

Use ECC if you can.  It depends on motherboard and processor.

If you insist on GELI, get an i5 or something else with AES-NI.  GELI will still suck the life out of it, but at least it will run reasonably fast.  I bought AMD for years, but they just don't seem competitive with the latest Intel stuff.

If you want low power and cheap, the lowest-end Sandy Bridge Celerons are great.  Surprisingly fast, too.

I would pick one of the higher efficiency power supplies, 80+ gold.  They are not much more expensive and should be able to pay for the difference in a year or so.  Been doing some research on that, and the Rosewill Capstone supplies are supposed to be surprisingly good.  The modular 450W is $70 right now, but has been on sale for less.  I'm tempted.  The other choice would be the Seasonic SSR-360GP.  Try to size it so the idle drain is at least 20% of the power supply max to get the best efficiency.


----------



## Uniballer (Jan 29, 2013)

I agree completely with wblock, except that I sleep much better if I install ECC memory in the mission critical systems that need to keep running (environmental control system, telephone PBX, etc.).  I don't worry so much about my personal desktop/file server.  If it crashes I reboot it: minor inconvenience.


----------



## Sebulon (Jan 29, 2013)

IÂ´m still paranoid about having ECC everywhere, as much as possible. Bitflips can seriously damage the data you store, and the storage wonÂ´t even know, nor care about it. Maybe youÂ´ll end up shelling out a couple of extra bucks but youÂ´ll sleep better knowing your data is safe.

/Sebulon


----------



## clpollock (Jan 29, 2013)

I also make a point of using ECC to try to minimize the risk of data corruption. Some people think the risk isn't worth worrying about, but I prefer not to take chances. I use ZFS for the same reason.

ECC support on cheaper processors is perhaps the only remaining advantage of AMD. There are some problems, however:

1) A lot of motherboards apparently don't support ECC, even if the chipset and CPU do. Asus boards often do, but make sure you check the manual to be sure - it should have a section on configuring ECC.

2) AMD chips use more electricity than the Intel equivalents.

3) It's getting hard to find a good AM3+ motherboard with integrated video and ECC support, which would force you to install a video card. This would probably increase power consumption even more, even if you have no monitor attached. I don't think newer APUs with integrated video support ECC  unfortunately.

My own home server (used mainly to store photos and video) is currently using an aging single-core AMD chip with 4GB of RAM. It also has only 2 PCI Express slots, both in use, so I can't add a USB3 card. I've been thinking of upgrading it.

The most likely configuration would be a Xeon 1225v2, 8GB of ECC RAM, and one of these motherboards:

http://www.asus.com/Motherboard/P8C_WS/

http://www.supermicro.com/products/motherboard/Xeon/C216/X9SAE-V.cfm

I could build an AMD system for perhaps $200 less (I'm in Australia), but it would be less powerful and use more electricity. The above server/workstation class motherboards are perhaps also more reliable than a "consumer" grade AMD motherboard.

I eventually decided to make do with my current setup for another year or two.


----------



## J65nko (Jan 29, 2013)

If you plan to use ECC memory http://cr.yp.to/hardware/ecc.html could be an interesting read


----------



## Sebulon (Jan 29, 2013)

http://lambda-diode.com/opinion/ecc-memory


> A system on Earth, at sea level, with 4 GB of RAM has a 96% percent chance of having a bit error in three days without ECC RAM. With ECC RAM, that goes down to 1.67e-10 or about one chance in six billions.



Says it all really.

/Sebulon


----------



## naguz (Jan 29, 2013)

clpollock said:
			
		

> 1) A lot of motherboards apparently don't support ECC, even if the chipset and CPU do. Asus boards often do, but make sure you check the manual to be sure - it should have a section on configuring ECC.
> 
> 2) AMD chips use more electricity than the Intel equivalents.
> 
> 3) It's getting hard to find a good AM3+ motherboard with integrated video and ECC support, which would force you to install a video card. This would probably increase power consumption even more, even if you have no monitor attached. I don't think newer APUs with integrated video support ECC  unfortunately.


I have been looking at the Asus M5A78L-M/USB3 which is still availible here (Norway). But the selection is small, that is true (and sad)

@wblock: Is GELI really that CPU intensive? I've used luks on linux wuthout much of a performance hit even on laptops with old ULV-processors. Granted, they never moved that much data around... 

Going for a C216-chipset Mobo AND a Xeon breaks my budget quite a bit, sadly. But there is another thread on these forums which says the Celeron G540 Sandy Bridge) actually has EC support if paired with a supported intel server chipset. Does anyone know of other cheap-but-sufficient intel processors this is true for? I can't seem to find a list. I might spend a bit more on a server grade mobo, but not if I have to pair it with an Intel.

That said, I don't need super-low power for economy I just want it to run "cool enough" enough when it's semi-idle, which it will be most of the time. An AMD processor will probably be fine.


----------



## wblock@ (Jan 30, 2013)

GELI really does make a big difference, but whether it will be a problem depends on your usage.  One thing on my list to try is the built-in encryption on notebook drives.  That might be as fast as unencypted, but not all drives support it and I don't know how it can be used on a desktop.  And of course it limits drive choice.


----------



## jb_fvwm2 (Jan 30, 2013)

I've used geli several times, but each time it unexpectedly, after a few weeks,  refused the valid passphrase with some terse message. Was good as long as it worked...  Several ports have similarities to it and one can use them on part of the filesystem, after practicing enough to understand enough of the usages.


----------



## clpollock (Jan 30, 2013)

naguz said:
			
		

> I have been looking at the Asus M5A78L-M/USB3 which is still availible Going for a C216-chipset Mobo AND a Xeon breaks my budget quite a bit, sadly. But there is another thread on these forums which says the Celeron G540 Sandy Bridge) actually has EC support if paired with a supported intel server chipset. Does anyone know of other cheap-but-sufficient intel processors this is true for? I can't seem to find a list. I might spend a bit more on a server grade mobo, but not if I have to pair it with an Intel.


Here in Australia, it looks like you can get a Xeon 1225v2 for around $240, and the cheapest core i3 is around $115. A difference of $125 doesn't seem too bad, considering that the Xeon is a far more powerful processor, and is guaranteed to have ECC support. The difference may be more in Europe, but the Xeon might well be cheaper in the long run, since it will last you longer before you feel the need to upgrade again.


----------



## clpollock (Jan 30, 2013)

J65nko said:
			
		

> If you plan to use ECC memory http://cr.yp.to/hardware/ecc.html could be an interesting read


An interesting read indeed.

I also find in extremely annoying that some manufacturers seem to think their motherboard "supports ECC" if it will run with ECC memory installed, even if it doesn't actually enable the ECC functionality.

The Asus AMD motherboards that I've been using for the last 4 years have specific "ECC Configuration" sections with detailed options for setting the level of background scrubbing, etc. so hopefully they really do support ECC.

When I decided to use a socket 2011 Xeon for my latest workstation, ECC support was a lot harder to come by. Although it's expensive in Australia, I decided to use this Supermicro motherboard to remove all doubt:

http://www.supermicro.com/products/motherboard/xeon/c600/x9sra.cfm


----------



## Crivens (Jan 30, 2013)

When I pondered the question of spending a little more on memory to get ECC, our admin showed me a log file from one of the servers which contained the ECC entries. The rate of three days with 4GB looks accurate, I think, and when that bit destroys some data structure which ZFS needs, you may have all the redundancy you want. If the FS is buggered on the high level, you can loose it all. So I spent some bucks more to have that peace of mind.

The Asus AMD boards indeed have a good support for ECC in the BIOS, but do not think that BIOS is meant for anything other than Windows. I rarely had as much problems with ACPI, which has put me a bit off that brand.


----------



## von_Gaden (Jan 30, 2013)

With all my respect to reliability of ECC I'd note that if the error rate of non-ECC RAM is correct we should disbelieve and discard any data provided by devices with non-ECC RAM...
I'd like to remind you an ancient and still used method of parity check. It's done by the memory module (hardware) and guarantees 100% accuracy in 1 bit alteration.
Do you remember 201 error during IBM/PC computers POST (about 20 years ago)?
Non-ECC can't correct the errors but surely would detect them and lead to system halt.


----------



## naguz (Jan 30, 2013)

clpollock said:
			
		

> An interesting read indeed.
> When I decided to use a socket 2011 Xeon for my latest workstation, ECC support was a lot harder to come by. Although it's expensive in Australia, I decided to use this Supermicro motherboard to remove all doubt:
> 
> http://www.supermicro.com/products/motherboard/xeon/c600/x9sra.cfm


The cheapest Xeon+Mobo combo availible here, is some 480USD. So it seems I'll go with an Asus M5A78L-M/USB3 with an AMD FX-4100, which also has AES Extensions. This would total at about 225USD. At less than half the price, it seems the logical way to go, especially since I won't have to have a discrete GPU.


----------



## Crivens (Jan 30, 2013)

von_Gaden said:
			
		

> With all my respect to reliability of ECC I'd note that if the error rate of non-ECC RAM is correct we should disbelieve and discard any data provided by devices with non-ECC RAM...


Indeed we should, at least take the results with a grain of salt.

Parity might at least halt a system, but then again, is it used at all? According to anecdotal evidence simple parity memory is no longer in wide use, so I would not bet on that to work out. Also, in order to be seen at POST, the memory error must be detected at memory check during boot up, but the times where the BIOS did a real memory check for the full range are gone, aren't they? Today the OS would need to stop and display some kind of error message, but I have not seen them. So if these would be active today, you should have a pretty good chance to see one when you use a standard laptop with, say, 16GB memory during a trans-atlantic flight. Radiation which can cause these flips is a lot harder in that altitudes, you should be almost certain to stop the machine.

Running memtest for the whole flight may show the errors even in absence of ECC/parity, but I'd wager that would get you unwanted attention, staring at a screen where cryptic number patterns are scrolling by.


----------



## c_geier (Jan 30, 2013)

naguz said:
			
		

> Going for a C216-chipset Mobo AND a Xeon breaks my budget quite a bit, sadly. But there is another thread on these forums which says the Celeron G540 Sandy Bridge) actually has EC support if paired with a supported intel server chipset. Does anyone know of other cheap-but-sufficient intel processors this is true for? I can't seem to find a list. I might spend a bit more on a server grade mobo, but not if I have to pair it with an Intel.
> 
> That said, I don't need super-low power for economy I just want it to run "cool enough" enough when it's semi-idle, which it will be most of the time. An AMD processor will probably be fine.



All C216 chipset boards should support ECC RAM with Pentiums and Celerons, for example Supermicro X9SAE, check the supported CPUs.

The latest Opterons 33XX support ECC RAM, too and matching boards with ECC support might be a bit cheaper (for example the ASUS M5A99FX PRO R2.0 can be had for about 130 Euros), but I couldn't find any actual power uasge figures yet.


----------



## wblock@ (Jan 31, 2013)

naguz said:
			
		

> The cheapest Xeon+Mobo combo availible here, is some 480USD. So it seems I'll go with an Asus M5A78L-M/USB3 with an AMD FX-4100, which also has AES Extensions. This would total at about 225USD. At less than half the price, it seems the logical way to go, especially since I won't have to have a discrete GPU.



As long as it runs console only.  Xorg on FreeBSD does not support those GPUs yet.

Update: that says it has a "ATI Radeon 3000", which actually should work.  It's the APUs that are not supported.


----------



## naguz (Jan 31, 2013)

c_geier said:
			
		

> All C216 chipset boards should support ECC RAM with Pentiums and Celerons, for example Supermicro X9SAE, check the supported CPUs.
> 
> The latest Opterons 33XX support ECC RAM, too and matching boards with ECC support might be a bit cheaper (for example the ASUS M5A99FX PRO R2.0 can be had for about 130 Euros), but I couldn't find any actual power uasge figures yet.


The Pentiums as Celerons aren't very state-of-the-art though. While I know nothing of their real performance, I'd guess they're generally below an i3. Also, are you sure about that claim?

I could go for an Opteron with the Asus M5A78L-M/USB3 AFAIK, but would it matter much compared to the FX one?

@wblock@: I've never ran anything more graphical than ncurses with FreeBSD. God knows its hard enough to maintain ports with WITHOUT_X11 set...


----------



## wblock@ (Jan 31, 2013)

I have a Celeron G530 in a server.  Kind of a nice little processor, what I hoped the Atom would be.  Cheap (under $50, under $35 on sale), low-power, about 85% as fast as an i3 according to cpubenchmark.net.  With an SSD or a fast disk, it's great.  With an ordinary disk, it's fine.


----------



## c_geier (Jan 31, 2013)

naguz said:
			
		

> The Pentiums as Celerons aren't very state-of-the-art though. While I know nothing of their real performance, I'd guess they're generally below an i3. Also, are you sure about that claim?



I haven't tried it myself, but to quote supermicro:

The X9SAE Motherboard Series supports a single IntelÂ® XeonÂ® E3-1200 v2 series,
Xeon E3-1200 series, PentiumÂ®, and CeleronÂ® processor in an LGA 1155 socket.
Leveraging the features of the Intel C216 chipset, the X9SAE motherboard provides substantial enhancement to system performance high performance gaming
platforms and entry-level workstations.​
note: i3 is not supported.

While Celerons and Pentiums might not have AES-NI and some other features (enabled?), the Ivy Bridge Celerons & Pentiums are of the same generation as the Xeons.


----------



## BlueCoder (Jan 31, 2013)

The original post is like something I posted somewhere else a year or so ago. Haven't yet built it.

The 2011 xeon processors are quite impressive for idle power and top computing muscle. 

But lately I'm been thinking about cores and how I really don't like "hyperthreading". It seems to me a properly coded scheduler with a few extra instructions from the processor could do that same thing as hyperthreading. An instruction such as "in case you stall preload this thread and run it ahead of schedule" and then balance the priorities to compensate. The bulldozer approach seems better to me where two cores sharing a floating point unit, would be even better if the FPU's were in a pool that could be used by all the processors. Nowadays I don't see the FPU being all that necessary, the graphics card has become the new FPU.

But the motherboards seem to be lacking for AMD processors. I prefer all x16 slots with x16 and about 3 8x.  I don't need sata 3 for single hard drives, the throughput from the hard drives doesn't max out what current drives are capable of.  Sata 3 and usb3 might be nice for an external drive array but I'd give them up for the power savings. But I definitely want ECC. I wouldn't even mind registered memory. It may be a bit slower but if I can put a ton of it in the system I think that would more than make up for it. The truth is the box will sit idle 98% of the time. I don't care if it burns twice as much electricity as other processors when it's maxed out so long as it can keep up. What can I say, I really want one of those 16 core engines.


----------



## throAU (Feb 1, 2013)

BlueCoder said:
			
		

> But lately I'm been thinking about cores and how I really don't like "hyperthreading". It seems to me a properly coded scheduler with a few extra instructions from the processor could do that same thing as hyperthreading.




2 things that will make that hard:


The CPU may turn cores off and on as required due to power constraints or to optimise multi-thread vs single thread performance

CPUs are designed for existing software.  Your funky scheduler will not necessarily speed up existing software


----------



## naguz (Feb 12, 2013)

Looks like I'm getting my hands on a PowerEdge t310 with an Intel Xeon 3400 series processor and 12gigs of memory - for FREE.  That's quite a "deal". It needs disks, a GPU and probably some modifications for noise/cooling as it will be standing in my living room.

Anyone knows if it is possible to cram the hardware into a normal ATX case?


----------



## xibo (Feb 15, 2013)

Many Intel Xeon mainboards use the SSI form factor, which is somewhat larger than ATX, more like Extended ATX.


----------



## naguz (Feb 16, 2013)

Nah, seems like it won't fit in my old case, which is a pity, as the Fractal Design case I have is pretty good noise-wise, and I could live well without hot-swap disks where I have to pull out the front panel anyway.

Still, the fan is not LOUD on low RPMS, but my God, when it spins up, it's a vacuum cleaner! Felt like sitting beside an AC in high mode.  Luckily, it only does that if the side panel is removed (for some reason). Hopefully it won't start to spin up even when I use it and it gets some load. I'm considering swapping out the CPU cooler anyhow, if I can fit a Noctua NH-U12 or similar to it somehow, Then I'll probably swap out the case cooler as well, or even remove it, have the CPU cooler blow out of the case, and have a separate fan on the HDDs.

Spent *hours* today getting the bloody thing updated, BIOS, iDRAC and whatnot, seeing as it hadnÃ¦t been done in ages, and so the built in update system wasn't working, and required a couple of pre-update-updates. Certainly not an ordinary desktop, this!


----------



## Terry_Kennedy (Feb 16, 2013)

von_Gaden said:
			
		

> With all my respect to reliability of ECC I'd note that if the error rate of non-ECC RAM is correct we should disbelieve and discard any data provided by devices with non-ECC RAM...


One thing to note in the quoted references is the age of the articles. For example, djb says "a typical computer with 256MB".

I remember many claims that we were reaching a limit to memory capacity (due to shrinking silicon feature size). The one that generated the largest hooplah was the often-repeated "any decrease in feature size will not work because the ceramic IC packages we use will emit so many alpha particles that the memory will be useless".

I've used just about every memory technology there is, from Williams tube (but that system was an antique at the time), mercury delay line (likewise), drum, core, Twistor, static and dynamic semiconductor RAM.

All of the server-class systems I specify have ECC memory, and log correctable / uncorrectable errors. As far as I know*, with over 300GB of ECC memory scattered among a dozen or so systems, I've only had one correctable error in the last year.

* Famous quote: "As far as we know, our computer has never had an undetected error."


----------



## naguz (Feb 17, 2013)

Damn it. Seems like it's the X3430 processor inside, which does NOT have the AES instruction set. That was really quite a disappointment... How much toll would GELI take on this 2,4Ghz Xeon?
Seems like I can't find a Xeon with AES-instructions for the 1156 socket at all. Don't wanna give it up yet, might have to test it and see how it goes.


----------



## Nukama (Feb 24, 2013)

Use ECC RAM, you then get some cryptic messages from time to time. (if you're unlucky)


```
[CMD="#"]cat MCA.txt[/CMD]
MCA: Bank 4, Status 0x9422400121080813
MCA: Global Cap 0x0000000000000105, Status 0x0000000000000000
MCA: Vendor "AuthenticAMD", ID 0x10ff0, APIC ID 0
MCA: CPU 0 COR BUSLG Source RD Memory
MCA: Address 0x5f311e70
[CMD="#"]mcelog < MCA.txt[/CMD]
HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
MCE 0
CPU 0 4 northbridge TSC 108d22803cdc 
ADDR 97f350 
  Northbridge RAM Chipkill ECC error
  Chipkill ECC syndrome = 2144
       bit32 = err cpu0
       bit46 = corrected ecc error
  bus error 'local node origin, request didn't time out
             generic read mem transaction
             memory access, level generic'
STATUS 9422400121080813 MCGSTATUS 0
MCGCAP 105 APICID 0 SOCKETID 0 
CPUID Vendor AMD Family 15 Model 31
```

Still waiting for the first laptop with ECC RAM to come.


----------

