# ZFS native encryption support



## lockdoc (Feb 23, 2012)

Is there any work done to port the zfs native encryption to FreeBSD?

The only information I could find so far
http://2007.asiabsdcon.org/papers/P16-slides.pdf


> per-filesystem encryption
> â€¢ soon



http://2007.asiabsdcon.org/papers/P16-paper.pdf


> Data encryption is a work in progress [Moffat2006]



Both seem a bit old, anywhere where *I* could find more information?


----------



## gkontos (Feb 23, 2012)

That would require ZFS v30. As far as I am aware Oracle has not released the code under CDDL.


----------



## phoenix (Feb 24, 2012)

ZFSv28 is the latest opensource version of ZFS.

ZFSv30 or ZFSv31 include encryption support, and are only available in Solaris 11.  None of the ZFS bits have been opensourced.  Thus, there's nothing "to port".


----------



## lockdoc (Feb 24, 2012)

So if there is no new open source release, what will happen to zfs on freebsd FreeBSD?

And in terms about the encryption they could work together with the guys/girls from Illumos? As far as *I* can see, they have it up on a todo list: https://www.illumos.org/projects/zfs-crypto


----------



## throAU (Feb 24, 2012)

As I undestand it, ZFS is forked?

Pretty sure the OpenIndiana/Illumos guys are working on ZFS independently of Oracle now?  As above, I suspect any improvements made by either Illumos or FreeBSD will be shared between them.


----------



## Sfynx (Feb 24, 2012)

What will this do with on-disk format compatibility? I always loved the fact that I could always import any pool version into any operating system as long as the supported pool version was equal or higher, even across little-/big-endian systems.

I guess this could still be possible with our own implementation if Oracle at least documents the on-disk format properly, else it'd be a reverse-engineering job which I'd not feel confortable with for enterprise storage purposes :/


----------



## lockdoc (Feb 24, 2012)

Or it won't be a reverse engineering an*d* there will be two different versions for zfs from now. Would be a pity.


----------



## fluca1978 (Feb 24, 2012)

I don't like the Oracle way of doing, and it did not surprise me that after having kicked off OpenSolaris also ZFS is going to become close sourced.


----------



## phoenix (Feb 24, 2012)

Search the forums.  This topic (future of ZFS) has come up many times already, and all the details regarding the ZFS Working Group have been listed in there.


----------



## overmind (Feb 29, 2012)

You could always use *ZFS* + *Geli*.


----------



## lockdoc (Mar 2, 2012)

overmind said:
			
		

> You could always use *ZFS* + *Geli*.


I am using it, but once the pool grows with lots of HDDs it becomes pretty slow. Geli is working under zfs and there are too many _extra_ operations needed. The ZFS encryption method is directly in the i/o chain (compression->encryption->checksum->deduplication) implemented, which *I* suspect to be much faster.


----------



## elfsechsundzwanzig (Mar 2, 2012)

lockdoc said:
			
		

> I am using it, but once the pool grows with lots of hdd's it becomes pretty slow.
> Geli is working under zfs and there are too many _extra_ operations needed.
> The ZFS encryption method is directly in the i/o chain (compression->encryption->checksum->deduplication) implemented, which i suspect to be much faster.



That might be true, but unless Oracle changes its mind (which, I guess, is highly unlikely), it seems like the only option.


----------



## lockdoc (Mar 2, 2012)

If I am not mistaken there are a couple of others also affected by this among FreeBSD such as OpenSolaris, OpenIndiana and Illumos. So this might be the chance for those 4 to group up on ZFS development


----------



## elfsechsundzwanzig (Mar 2, 2012)

Yes, that's possible. Instead of hoping that Oracle will open the sources of ZFSv30, one can gather some able people and try to implement zfs-native encryption support using the codebase of the v28..  

I would really appreciate it


----------



## gkontos (Mar 2, 2012)

lockdoc said:
			
		

> If I am not mistaken there are a couple of others also affected by this among FreeBSD such as OpenSolaris, OpenIndiana and Illumos. So this might be the chance for those 4 to group up on ZFS development



OpenSolaris is RIP thanks to Oracle. OpenIndiana is based on Illumos an OpenSolaris fork and FreeBSD gets its ZFS code from Illumos.

The only open source full production OS that supports ZFS is FreeBSD.


----------



## elfsechsundzwanzig (Mar 2, 2012)

And wouldn't it be great to have a FreeBSD with ZFS with native encryption support?


----------



## Zare (Mar 2, 2012)

I'm still looking for an in-depth benchmark/analysis of ZFS atop GELI devices. I don't have enough practical experience with ZFS to do that myself.


----------



## phoenix (Mar 2, 2012)

lockdoc said:
			
		

> If I am not mistaken there are a couple of others also affected by this among FreeBSD such as OpenSolaris, OpenIndiana and Illumos. So this might be the chance for those 4 to group up on ZFS development



Search the forums for "ZFS Working Group".  This is already happening.


----------



## phoenix (Mar 2, 2012)

gkontos said:
			
		

> OpenSolaris is RIP thanks to Oracle. OpenIndiana is based on Illumos an OpenSolaris fork and FreeBSD gets its ZFS code from Illumos.
> 
> The only open source full production OS that supports ZFS is FreeBSD.



And Nexenta.  And several Linux distros via ZFS-on-Linux.  And several others.  ZFS is out there, and many OSes/projects/groups/people are using it.


----------



## gkontos (Mar 2, 2012)

phoenix said:
			
		

> And Nexenta.  And several Linux distros via ZFS-on-Linux.  And several others.  ZFS is out there, and many OSes/projects/groups/people are using it.



I did mention the word "production OS"


----------



## phoenix (Mar 2, 2012)

How is NexentaCore not a production OS?

How is a Linux distro not a production OS?


----------



## gkontos (Mar 2, 2012)

phoenix said:
			
		

> How is NexentaCore not a production OS?



I was under the impression that it has been replaced by NexentaStor which is not a open source OS



			
				phoenix said:
			
		

> How is a Linux distro not a production OS?



Not when it comes to native ZFS support. Meaning that there have been enough installations followed by testing backed by a team that can handle bug fixes and development, giving the green light for production usage. 

I don't like to play with words and I am a big ZFS advocate. But I believe that the only open source OS which can fully support ZFS for production today is FreeBSD.

I also have nothing against Linux when it comes to ZFS integration and sincerely express my support to the ZFS Working Group


----------



## fgordon (Mar 31, 2012)

I'm using Geli (AES 128) + ZFS (freebsd FreeBSD 8)  with raid-z2 and 12 x 2 TByte SATA (SII 3114 PCI) CPU AMD X2 3800 with 3 Gbyte RAM so a rather "low end" server now 

I get between 30 to 35 Mbyte / sec over SMB export if I copy large amounts of data - as I'm away from my server I cannot benchmark at the moment.

All in all I think GELI+ZFS already offers quite a nice performance.

Because of the time consuming crypt operation (at least for Non AES NI) I'm sure the difference between geil + zfs and native crypto zfs is minimal - ZFS native crypt is optimizing the 1% CPU usage, while 99% percent of the CPU time is still consumed by (de)crypting algos.


----------



## none (Apr 16, 2012)

I also have performance problems using Geli on ZFS. I run an Atom from Soekris (1.6GHz and 2GB RAM), and my test case uses 4 160GB SATA disks. I get around 2~3MB/s. AES 128 bits.

I was hoping for something more fast, as this same box can serve up to 10MB/s without Geli.

*I*f there is any option here, I'd really like to know 

*T*hanks,

none


----------



## throAU (Apr 17, 2012)

If you want encryption performance, surely you want a CPU with AES encrypton in hardware.  I don't think an atom is ever likely to give you this, it's simply not the right hardware for the job.


----------



## ondra_knezour (Apr 17, 2012)

Some sort of hardware cryptoaccelerator may help you, something like http://soekris.com/products/vpn14x1/vpn-1411.html


----------



## none (Apr 17, 2012)

throAU said:
			
		

> If you want encryption performance, surely you want a CPU with AES encrypton in hardware.  I don't think an atom is ever likely to give you this, it's simply not the right hardware for the job.



I know it*'*s NOT the perfect hardware for the job, but it*'*s for a home, three accesses at a time tops, and is what I have here now. I'd like it to be low power demanding, so a Soekris is great for me. I don't want to make it push GB/s, just I'm trying to look for the best way to do it on my hardware.

ondra_knezour: I have one of those on the other Soekris, but unfortunately there is no version for the net6501. And the 5501-70 has too little memory (512MB). Thanks for the idea 

none


----------



## fluca1978 (Apr 17, 2012)

throAU said:
			
		

> If you want encryption performance, surely you want a CPU with AES encrypton in hardware.  I don't think an atom is ever likely to give you this, it's simply not the right hardware for the job.



I agree, sounds like you are asking your hardware more than it can do!


----------



## fgordon (Apr 18, 2012)

Via Nano is a similar CPU to Intel's Atom - and most of the Via CPUs come with padlock = hardware crypt.

e.g. http://www.via.com.tw/en/products/processors/nanoX2e/index.jsp

At least with Linux the speed of my Via Nano 24/7 server is quite nice (AES 256 bit) probably FreeBSD will support padlock too.


----------



## none (Apr 18, 2012)

fgordon: thanks. But my issue is that I already have the box.

*A* bit of an update. I'm trying the same config using a different machine. This time it's a server:


```
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RELEASE #0: Wed Mar 21 19:25:12 BRT 2012
    root@macgyver:/usr/obj/usr/src/sys/net6501-amd64 amd64
can't re-use a leaf (geom_label)!
module_register: module g_label already exists!
Module g_label failed to register: 17
CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2327.55-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x10676  Family = 6  Model = 17  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33056464896 (31525 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
```

I have plenty CPUs and RAM (32GB). *T*he disks are SAS 146GB. And after all I still have less them 10MB/s. A file copied from NFS got me 9MB/s.

I'm not complaining about a thing, as I said before my goal is to find the best way to use my hardware and FreeBSD and ZFS with crypto. Apart from a CPU that have AES-fu, is there any other thing that could make it faster?

*T*hanks,

none


----------



## phoenix (Apr 18, 2012)

Skip the crypto, and things will speed up *a lot*.  

Thus, you need to find a crypto-accelerator.  The simplest method being to use a CPU that includes it (Via Padlock, Intel AES-NI, etc).

Also note that GELI is single-threaded, and thus only uses a single CPU core; compared to ZFS which is multi-threaded and will use as many cores as possible for compression, dedupe, interrupts, etc.

Doesn't matter how fast your disks/RAM are, as GELI is the bottleneck.


----------



## Crest (Apr 19, 2012)

GELI is not single threaded. GELI will start one thread per logical core per GEOM provider by default. GELI will scale quite nicely. Just throw away the crappy Intel Atom based hardware. My Phenom 2 X6 1090T based desktop reaches > 200MiByte/s with a four disk RAID-Z1 on top of four GELI providers on cheap 5900rpm SATA disks. GELI is the difference between 220MiByte/s and 250MiByte/s. The CPU load at full throughput is < 40% spread more or less equally over all cores. ZFS needs enough RAM and bit of CPU time. GELI needs a lot of CPU time. Neither ZFS nor the in kernel AES implementation where designed for under powered Atom boxes. The same zpool saturates the 1.3GHz AMD K10 dual-core CPU on a HP N36L at 45MiByte/s.


----------



## phoenix (Apr 19, 2012)

Crest said:
			
		

> GELI is not single threaded. GELI will start one thread per logical core per GEOM provider by default.



Good to know.  Thanks for the update!


----------



## none (Apr 19, 2012)

*T*hanks both for the info. As I said, I already have the atom box and changing is really painful. But is there a test on how the Via performs?

I need a low power server, so must be Via. The problem is to know how it performs to figure out if selling the atom and buying a Via board (that is not an easy thing in here) will pay off.

*T*hanks for all,

none


----------



## throAU (Apr 19, 2012)

Just with regards to your server test:  I don't think that Xeon includes AES-NI either, so it will be similarly slow (compared to a hardware accelerated AES device / CPU).  It's a much faster CPU yes, but still running AES in software.

The AES instructions in the Core I series (for example) speed up AES by a factor of about 20-30 over doing it on a Core 2 (which itself is more powerful than an atom by quite a margin).

Using AES on your Atom may be fine for securing low-bandwidth WAN connections, but trying to run disk IO through it, it simply won't keep up - you're just asking too much of it.


----------



## sim (Jun 4, 2012)

A fellow ZFS/GELI/Atom homeserver user here...

In light of the assertions above indicating that the Atom should be junked for this workload, I've just run some simple tests, timing the how long it takes to write various things to my GELI'd Zpool.

This is by no means a comprehensive HD test, just a quick look to get some indications of the encryption overhead. I've only looked at large transfers where the onus is on encryption and writing rather than seeking.

Copying a 701MB file file from an unencrypted zpool to a GELI'd zpool takes on average between 10-15 seconds. (Both single-drive pools are identical 2.5" 5400rpm 640GB jobbies).  I expect the read is at least partly from the ZFS L2ARC, but can confirm that the disc is physically active for the full duration of this test.

Writing 1GB of /dev/zero to the encrypted pool takes on average 15 seconds.

Again, I know these are not rigorous benchmarks! 

In any case, for home use it works just fine.

I agree ZFS encryption would be very nice, and I look forward to the day that we may see it, but don't write off the alternatives too hastily.

sim


----------



## none (Jun 4, 2012)

sim,

I got the feeling that Soekris boards was the one to blame, as its buses are unable to handle the data flowing. A test on Intel D525MW got better results (will have numbers soon), and the CPU is only 200MHz faster. I moved the file server to this board, and will finish the migration soon. Ordered some sata stuff from Internet and when it arrives I will be able to make it the file server as the Soekris were (I had a PCIe eSATA card, and this card has no PCIe - just mini-PCIe. My old Sil3124 can't handle the port-multiplier well).

I read some things from Soren (Soekris Engineer), and the bus factor got me suspicious. Still a chance for ZFS and GELI 

Thanks,

none


----------



## bbzz (Jun 4, 2012)

Performance and low-power never go that well together. Depending on your idea of low-powered, new Xeon at 17W is very cool.


----------



## none (Jun 5, 2012)

bbzz said:
			
		

> Performance and low-power never go that well together. Depending on your idea of low-powered, new Xeon at 17W is very cool.



bbzz: that is more then I was expecting from that box. It is a good hint for sure, thanks. But my machine draws just 10w when idle now (the soekris) and the atom board draws around 24w all board. Unfortunately I don't know any board that would be as efficient 

thanks,

none


----------



## overmind (Jun 5, 2012)

none said:
			
		

> bbzz: that is more then I was expecting from that box. It is a good hint for sure, thanks. But my machine draws just 10w when idle now (the soekris) and the atom board draws around 24w all board. Unfortunately I don't know any board that would be as efficient
> 
> thanks,
> 
> none



To be accurate, from my measurements resulted that PCEngines uses 4W of power (without hard drive but if you use a laptop hard drive just add 1W more) and my atom board (an ASrock 1.6 dual core atom, 330D) uses a lot of power: 35W, without hard drive). What CPU uses that soekris? Is it Geode 500mhz like in PCEnines?

The new Xeon uses 17W of power just for the CPU. How much power will use the entire system with?


----------



## bbzz (Jun 6, 2012)

Not sure but it really is low for such powerful CPU. Also 17W is max, meaning you'll be idling at much lower than that. Rest of components shouldn't take nearly as much power.


----------



## none (Jun 8, 2012)

overmind said:
			
		

> To be accurate, from my measurements resulted that PCEngines uses 4W of power (without hard drive but if you use a laptop hard drive just add 1W more) and my atom board (an ASrock 1.6 dual core atom, 330D) uses a lot of power: 35W, without hard drive). What CPU uses that soekris? Is it Geode 500mhz like in PCEnines?
> 
> The new Xeon uses 17W of power just for the CPU. How much power will use the entire system with?



The Soekris I said uses an atom 1.6GHz. I measured it and with notebook HDD it draws 10w. Also measured the Intel d525mw using one notebook HDD and using picoPSU readings were 24w, normal HuntKey PSU was 36w.

Is there any board for Xeon energy efficient that is targeted for small vorm factors? A friend bought a VIA solution and said performance was great, limited by the HDD itself. He uses linux though. It's hard to find VIA hardware in here, nevermind specific Xeon configs. But it is always good to know there is the option.

*T*hanks,

none


----------



## MorgothV8 (Sep 9, 2012)

Netbook ACER Aspire - 1,6 GHz Atom.

ZFS:
Small boot with ZFS unencrypted (1G in size, about 180 Mb used)
Rest is 150 Gb ZFS on GELI ada0p4.eli

Write is about 2 Mb/s (megabytes not megabits)
Read is little more, but at maximum 3 Mb/s.

Quite slow, but this a test setup
FreeBSD 10 i386 CURRENT - compiled world 20.07.2012

Planning to use GELI on my home server - amd64 of course
Intel Core i5, disks SATA 3
System would be 10-CURRENT updated every month by build/install world/kernel

What should I expect on MIRROR of 2 x 2TB WD disks, 16 GB 1600 DDR3 RAM
zpool create zdata mirror ada1.eli ada2.eli (with propper 4K alignment)
Root of the system, and all system data (mean all data that can be restored: sources, ports, packages etc) would be on single SSD disk, about 80Gb:
zpool create zroot ada0
?? Any estimations ??


----------



## vermaden (Sep 9, 2012)

MorgothV8 said:
			
		

> Write is about 2 Mb/s (megabytes not megabits)
> Read is little more, but at maximum 3 Mb/s.


With which cypher and with how many bits?


----------



## MorgothV8 (Sep 9, 2012)

All defaults ..... it is/was a test installation.
I can search the manual, but I just used default options......


----------



## vermaden (Sep 10, 2012)

The default is 128-bit AES-XTS, You will get better read/write speed with 128-bit AES-CBC.


----------



## HarryE (Sep 10, 2012)

I 've got 40MB/s write, 60MB/s read from my ZFS RAIDZ 4 x 1TB HDD, AMD Athlon II X2 250 Dual Core, 3000 MHz, GELI default.
HTH


----------



## ondra_knezour (Sep 10, 2012)

I would expect dozens MB/s, as i5 CPUs have HW accelerated AES encryption, which can have huge impact on data throughput using encryption/decryption, search for HW SW AES benchmark for example.


----------



## Sebulon (Sep 10, 2012)

@MorgothV8

Here is my research on GELI/ZFS. The use of aesni.ko raises throughput to native. I suggest loading that, combined with AES-CBC 128 bit for optimal performance.

GELI Benchmarks

/Sebulon


----------



## MorgothV8 (Sep 10, 2012)

OK 40-60 MB/s with Athlon X2 without AES seems already quite fast - and with AES-HW I expect little more.... 
It is enough for me.
Thanks, when I finally try it, I'll post about results.


----------



## MorgothV8 (Dec 16, 2012)

Results are in some other thread, but they're about 160 MB/s.


----------

