# How to manage encryption keys with geli and ZFS?



## justinnoor (Mar 15, 2019)

Since this is not a hard technical question it was placed in off-topic. Please let me know if I should move it.

When using geli encryption on larger ZFS machines, it would seem practical to have all of the disks share the same master key so that the administrator would not have to enter a password for every single storage provider in the event of a reboot. Perhaps that would mean saving the master key to some external device, such as a USB stick. This could also make it easier to move disks to other machines if needed, since a USB stick can be attached to another machine on the fly.  Of course backups of the USB could be made in case one were lost or damaged.

Or maybe another alternative would be to have the same keyfile on every server, so that disks can be exchanged between machines. Every disk could share the same key. Maybe some sort of key rotation system could be implemented.

How are geli encryption keys systematically managed on larger ZFS machines/data centers?

Protection against expert cryptography is not the intent here. The main concern is protecting disks from landing in the wrong hands, i.e, if they were stolen or lost. They would at least have some degree of extra protection, right?


----------



## ralphbsz (Mar 15, 2019)

Wrong.  Or perhaps a little bit right.  But in computer security, "a little bit" doesn't cut it.

Your first step needs to be: Think about what you want to accomplish.  How important is your data?  How big is the damage if it gets revealed?  Will the economic cost to you be $100, $1M, or $10B?  If the answer is "small", then stop.  Don't waste your (and our) time by trying to implement something that most likely will have more holes than a swiss cheese.  If your answer is "big", then hire some experts to do it.  Your best bet at that point is probably to outsource your computing or at least its management to an experienced and trustworthy provider.  This will cost you something, but if your data is worth a few billion, then managing it will be worth some money.  In the middle is a painful grey zone, where it is desirable to protect your data, but it isn't valuable enough to invest lots of money into protecting it.

Second step: Think about likely attack vectors.  Who is after your data?  If you have made enemies of a major state actor (US, Russian or Chinese government as examples), then stop.  Nothing an amateur can do will protect you against a concerted attack by their law enforcement agencies.  Similarly, if you have secrets that are of national security importance, then you can safely assume that a non-existing agency with a name like [CN][IRS][AO] or similar alphabet soup is using "national technical means" (that's a term of art) to find out what you're doing.  What this leaves to guard against is commercial theft of data (usually by pretty clueless thieves), and preventing embarrassing mistakes. So are you worried about someone coming and grabbing your server?  Are you worried about someone hacking into your server?  Spying on the network connection to it (for example by using your unsecured wireless)?  Or are you _only_ worried about a disk going into the trash?

Encrypting disks helps against a very small number of attack vectors.  It uses either hardware encryption (commonly called "SED" or self-encrypting drives or T10 after the name of the INCITS committee that creates the SCSI and SATA encryption standards), or software encryption (geli is one example).  It helps when a disk has been separated from the system, and is found powered-down.  It does not help when the disk is powered up and in use, because then the channel is open, and an attack through the server is the easiest path.  It also only helps if the attacker is guaranteed to not get their hands on the key.  So you need physical security against an attacker getting to your running server, or physically stealing it together with whatever thing holds the key.

But: Using disk encryption only against people reading your disk after you throw it in the trash is difficult and inefficient.  There is a much simpler way to protect that: take a hammer to the disk before putting it in the trash.  So for the rest of the discussion, let's assume that security of discarded media is not relevant.

The security of a storage encryption system depends crucially on the security of the key.  If someone can get the key, the game is over.  I used to work on a system that was able to encrypt its disks, and our joke was that using hardware-encrypting drives is like closing the bathroom window in the barn after the horse has left.  Most horses vanish because the barn door is open, or because a horse thief opens the barn door which isn't locked.  Protecting against the bizarre attack that the thief crawls through the bathroom window and open the barn door from inside is only sensible after you have secured everything else, and in the case of storage, that's physical and network security.  What we conclude from that is: never store the key in an insecure location.  Putting it on a USB stick that's physically in the server at all times is just dumb.  It's like hanging the key to the barn door on a nail next to the door outside.  The correct place to store they key is on a system that's intrinsically more secure than the thing you are protecting.  For small systems, the best option is to use a human who has to enter the key, and the key is used once to unlock and then discarded.  That's a little bit secure, but inconvenient.  To make it more secure, use multiple factors (like text string password + touch key or fingerprint reader), and encrypt the key itself in transit (give the human an encrypted key store, those can be bought commercially).  For large systems, the only practicable solution is to implement a key management system which is itself physically secure, and produces keys when needed, using secure communication channels.  To figure out how to do that, read about Kerberos, and how one physically protects Kerberos servers (the original MIT ones were in special steel cages in the data center).  Key management for storage is big business (there is a handful of specialized companies that do it, plus groups in the big companies like EMC, IBM, Google, HP and so on); this is not something for amateurs.  By the way, I would never implement something like that myself either: while I have worked in such groups, it's way to complex and intense for a person to do themselves.

Your last question is: is it OK to use the same key on multiple disks?  In theory, no.  Security people always advise to expose a key as little as possible, to keep the attack surface small.  Everytime the key travels or is used, there is a risk of it being exposed.  In practice, compared to all other risks, the exposure of the key itself is minor, and sharing it among disks is the least of your problems.  Personally, my advice would be: Use geli, remember the key in your brain, do not store it in cleartext anywhere (not on a USB stick, not on a yellow sticky note attached to the computer), and type it in everytime the system boots.  Or look into the various hardware security devices (Yubikey, RSA tokens, Titan keys, ...) to handle the key. Or just put your data into a cloud provider, where it is much more secure than anything you can do yourself.


----------



## justinnoor (Mar 16, 2019)

Thank you, sir. The thoroughness of your response is greatly appreciated. My apologies for the hastiness and disorganization of my post. I need to do better.


----------



## Eric A. Borisch (Mar 16, 2019)

There is one case not covered in this excellent response: warranty service.

Using encryption with always-in-the-system but not on the same drive (different drive; USB stick) can provide ease of use (you can remotely reboot without needing to enter the password) but you can still pull a faulty drive and return it for warranty service without worrying about the data that may or may not still be readable on the drive.

Most vendors won’t replace a (physically) hammered drive. 

As clearly pointed above, this isn’t protection against determined hackers, but it isn’t obtrusive, and it provides a layer of confidence that your drive won’t be read once it is separated from your system.


----------



## thogard (Mar 16, 2019)

What scenario are you looking at?  If your only risk is making sure a removable disk isn't taken out of a computer by accident and handed to a recycling group that might leak the data, then the same key on all drives would be fine.  The disk being lost after an its upgraded to a bigger one?  A removed failing disk ending up in a pile of bad disks and being ignored? The whole machine being stolen? If you have tighter security requirements, find a expert.

You mention theft.  Is that theft out of a car when drives are taken from one location to another or every computer in the building gets stolen by random thieves or employee internal theft or industrial espionage?   The USB thing is fine with the 1st case but not the other 3.

If your servers are in a secure location where theft isn't an issue, a USB stick with the keys on it attached to a string to the rack could be enough. If your not worried about the servers being stolen, then an internal USB stick might be enough for you.  If its more complex than that, you will also want to look at the issue that a running encrypted drive doesn't look encrypted to the operating system since it decrypts it on the fly.


----------



## ralphbsz (Mar 16, 2019)

The warranty example is indeed a really interesting borderline case.  Interesting because we can use it as a "teachable moment" to understand what encryption guards against.

Let's assume that they physical server and its drives are in a secure location.  That means we are not worried about a burglar coming and taking the whole thing (with or without USB stick), nor a burglar or rogue employee taking an individual disk.  Let's also assume that the OS and network are also sufficiently secure, so we are not worried about an unauthorized hacker logging in and reading a file that should have been encrypted from them them.  In that case, what are we worried about?  Aren't we done?  No, the last remaining source of data being exposed is when the drives deliberately leave the site. One reason for leaving might be that they are being decommissioned: our old 4TB drives are too small, we bought 16TB ones to replace them, and afterwards the 4TB will go in the trash.  In that case, we can erase the 4TB drives, either with "dd if=/dev/zero of=/dev/adaXX", or with a hammer (in the former case, we can get $50 by selling the used drive, but at a very small risk that the drive is still readable by using special techniques).  The other reason is that a drive has died, and we need to ship it back to the vendor for warranty replacement.  In that case, the "dd" trick won't work, since the drive is dead.  But honestly: The drive is only worth $200.  Your data is probably worth way more, otherwise you wouldn't have done all that encryption stuff (which is a lot of work to set up, and constant hassle with keys).  So my personal answer would be: The $200 loss on the warranty is part of the cost of running a secure server.

By the way, I've worked for some of the big companies that use many disks (the likes of EMC, Dell, IBM, HP, Facebook, Amazon, Google, Apple, ...) and they have contracts with disk vendors like Seagate/WD/Hitachi.  Part of those contracts are that warranty is handled differently from normal consumer channels, usually without actually shipping dead drives back, unless they are needed for an autopsy to do failure root cause analysis.  And customers where data security is actually important (such as the big internet companies, nuclear weapon research labs, ...) all have policies and procedures that guarantee that no disk drive ever leaves the site without first having been mechanically shredded.

To thogard's point: While using a hardware-encrypted disk, the disk is "unlocked", and completely readable to the OS.  If someone can become root on the system, they can read the disk.  Encryption does not replace access control and authentication.


----------



## thogard (Mar 17, 2019)

ralphbsz said:


> The warranty example is indeed a really interesting borderline case. Interesting because we can use it as a "teachable moment" to understand what encryption guards against.



The warranty issue can be a major issue.  Its the reason we don't buy new mac minis as we can't remove the storage before returning them for repair which makes them disposable just like hard drives.  We have a good relationship with our hard drive supplier so that if they don't have to return a warrantee drive, we don't have to give it to them but any drive less than about year old will need to be returned.  Simple encryption fixes that problem.

If you use simple encryption, make sure you use a good random key.  Hard drives are full of sectors that are well known so make sure your key isn't searchable.  One issue with cracking encryption isn't so much guessing the right key, but being able to verify you have the correct key once you guess it.


----------



## Eric A. Borisch (Mar 17, 2019)

For some instances (like home users), $200 may not be insignificant when the drive is under warranty and could otherwise be replaced for free.

In those cases, such a setup (separate volume storage of keys with automatic decrypt on boot) provides minimal overhead and complexity while providing a tangible benefit (significantly reduced worries about sending your defective drive with personal data back to the manufacturer.)

Perfect is the enemy of good.


----------



## D-FENS (Mar 17, 2019)

justinnoor said:


> Since this is not a hard technical question it was placed in off-topic. Please let me know if I should move it.
> 
> When using geli encryption on larger ZFS machines, it would seem practical to have all of the disks share the same master key so that the administrator would not have to enter a password for every single storage provider in the event of a reboot. Perhaps that would mean saving the master key to some external device, such as a USB stick. This could also make it easier to move disks to other machines if needed, since a USB stick can be attached to another machine on the fly.  Of course backups of the USB could be made in case one were lost or damaged.



The disks could have different keys indeed. This is how I like to do it. Entering password once is not a problem, because the keys can be stored on an encrypted file container that can be mounted with a single password to a ramdisk via mdconfig. Then all the keys can be read from the ramdisk and the ramdisk can be destroyed.


```
echo "Unlocking encrypted storage."
mdconfig -a -t vnode -f /path-to/encrypted-file-container.vol -u 200
geli attach /dev/md200
mount -o ro /dev/md200.eli /mnt/keys

geli attach -p -k /mnt/keys/hdd0.key gpt/hdd0
geli attach -p -k /mnt/keys/hdd1.key gpt/hdd1
...
geli attach -p -k /mnt/keys/hdd500.key gpt/hdd500

umount /dev/md200.eli
geli detach /dev/md200
mdconfig -d -u 200
```

Edit:
By the way, I forgot to mention how to create the encrypted file container:

```
truncate -s 4M hdd-keys.vol
dd if=/dev/random of=./hdd-keys.vol bs=4M count=1
mdconfig -a -t vnode -f ./hdd-keys.vol -u 200
geli init -e AES-XTS -s 4096 /dev/md200
geli attach /dev/md200
newfs /dev/md200.eli
```


----------



## justinnoor (Mar 17, 2019)

roccobaroccoSC said:


> Then all the keys can be read from the ramdisk and the ramdisk can be destroyed



But what happens if a disk is physically separated from the system? During this state, would the disk be encrypted?


----------



## ralphbsz (Mar 17, 2019)

Eric A. Borisch said:


> For some instances (like home users), $200 may not be insignificant when the drive is under warranty and could otherwise be replaced for free.
> 
> In those cases, such a setup (separate volume storage of keys with automatic decrypt on boot) provides minimal overhead and complexity while providing a tangible benefit (significantly reduced worries about sending your defective drive with personal data back to the manufacturer.)
> 
> Perfect is the enemy of good.


Agree: the geli solution is good enough to guard against the "attack" of sending a still functioning (i.e., readable) but defective (i.e., needs warranty replacement) drive away.  And as you say, it is low overhead (a few minutes to hours to set up).  It does not guard against a lot of other attacks, but that's a compromise the user has to make.  In this scheme, using a single key, and storing it right on the system (for example in a USB stick that's always plugged in, or like roccobarocco's example) is fine, as overuse and exposure of the key is not the problem.

You're perfectly correct that I've been striving too much for the "perfect" here.  That's because I've worked with people who do have nuclear secrets (as an example) on their computers.  For them, using encrypting drives is not the complete solution to their data security problem.  For them the solution has to also have other ingredients, such as making sure no piece of hardware (not even a cable) ever leaves the data center without first having been shredded, and having guards with machine guns at the door.  In such an environment, encrypted disks can be part of the solution.


----------



## D-FENS (Mar 17, 2019)

justinnoor said:


> But what happens if a disk is physically separated from the system? During this state, would the disk be encrypted?


Of course it would! The disk is encrypted with its own key. The key itself is located on an encrypted file container that needs to be unlocked with your password in order to access the disk keys.
So you can unplug the drive and send it for repair, the data is encrypted and unreadable for people that don't have the file container AND know the password to it.
You could put the encrypted file container on the disks itself (for example if you have an unencrypted boot drive, or a boot drive independently encrypted with password only), or on another drive - USB, HDD or whatever.


----------



## linux->bsd (Mar 17, 2019)

roccobaroccoSC said:


> Of course it would! The disk is encrypted with its own key. The key itself is located on an encrypted file container that needs to be unlocked with your password in order to access the disk keys.
> So you can unplug the drive and send it for repair, the data is encrypted and unreadable for people that don't have the file container AND know the password to it.
> You could put the encrypted file container on the disks itself (for example if you have an unencrypted boot drive, or a boot drive independently encrypted with password only), or on another drive - USB, HDD or whatever.



These are the kinds of practical solutions FreeBSD users come looking for on these forums. Doubt many people come here thinking they can prevent Chinese, Russian, or US government hackers from getting to them if they want to, so those types of debates seem incongruous here. Although the tools we suggest to others and use ourselves don't offer perfect protection, it's irritating watching people discourage others from using them -- they have their usefulness and application.


----------



## D-FENS (Mar 17, 2019)

One CAN prevent Chinese, Russian or US government hackers, but one needs to think on another level. Encryption is only one small part in the whole picture.
If the government wants to get you, they'll probably install bugs and sniff out your password via other sneakier means. Then they'll just image your drive when you're not at home and decrypt it in their lab.
Cracking the encryption is not necessary in those James Bond scenarios, because there are quicker and cheaper ways.

Also, there's always some unencrypted part on the boot drive. Be it /boot, or even the boot loader. They can be hacked to log your password and send it via the network even before the machine boots. So the government breaks into your house, images your disk and hacks the bootloader. Then you come home and type your password - and they have all your files.

The point is, security depends on what you want to be protected against. And good security is expensive, it includes software, hardware, building, people etc.
Disk encryption protects against leaking data due to lost or stolen disk and some basic physical attacks but not much more.


----------



## linux->bsd (Mar 17, 2019)

roccobaroccoSC said:


> One CAN prevent Chinese, Russian or US government hackers, but one needs to think on another level.



No, one _can't_. If the Pentagon itself can't prevent the Chinese government from sitting inside their network for months even after being discovered, you can't either. In addition to that, the NSA vacuums in all network traffic leaving the US border. If they want to devote resources to decrypting your traffic they can (and might even succeed).

The point is there's been a recent tendency to bloviate about security issues. People come here looking for a small bit of protection for their data -- they're not Edward Snowden trying to evade detection from world powers while on the lamb. Pointing out the limitations of what these solutions offer while still offering help on implementing them is the correct approach, not discouraging their use.


----------



## D-FENS (Mar 17, 2019)

Indeed. Unless one fears specific risk of being spied on, the risks we need to protect agains in general are: (1) automated mass attacks via the network, i. e. you need good firewall, (2) thieves trying to steal your notebook, i. e. disk encryption.
That covers you from lower 95% of the threats. And the last 5% are quite expensive to protect from. We need to take into account the value of the data we're protecting and not overdo it.

So in summary, the solution above seems appropriate for home use or a small business.


----------



## justinnoor (Mar 17, 2019)

roccobaroccoSC said:


> The key itself is located on an encrypted file container that needs to be unlocked with your password in order to access the disk keys.



Ah yes. I had to re-read that. I think I’m understanding this. This seems like a very practical solution. Much appreciated. I will brew on it.


----------



## ralphbsz (Mar 18, 2019)

linux->bsd said:


> The point is there's been a recent tendency to bloviate about security issues. People come here looking for a small bit of protection for their data -- they're not Edward Snowden trying to evade detection from world powers while on the lamb. Pointing out the limitations of what these solutions offer while still offering help on implementing them is the correct approach, not discouraging their use.


Guilty as charged.  I've been working for too long on big storage systems, including those used by people who have very serious security needs (and extreme performance and reliability needs).  When folks ask here on the forum "should I use or how to configure XXX to make my small system more secure (or faster or more durable)", I tend to tell them how hard it is to get to extremely good systems, and what pitfalls exist.  And forget that with 10% of the effort, they can get to 90% of the effect, and that is a good thing.  Feel free to point out my excesses when I go off again, I won't be offended.


----------



## linux->bsd (Mar 18, 2019)

ralphbsz said:


> Guilty as charged.  I've been working for too long on big storage systems, including those used by people who have very serious security needs (and extreme performance and reliability needs).  When folks ask here on the forum "should I use or how to configure XXX to make my small system more secure (or faster or more durable)", I tend to tell them how hard it is to get to extremely good systems, and what pitfalls exist.  And forget that with 10% of the effort, they can get to 90% of the effect, and that is a good thing.  Feel free to point out my excesses when I go off again, I won't be offended.



I chose not to comment because I noticed other forum members thanking you for your posts, which means they were gaining knowledge from the valuable information you were sharing. Your posts were indeed factually accurate and clearly gleaned from professional experience. They just seemed too discouraging. Battling to keep our companies' / clients' networks secure sometimes feels like we're standing on the edge staring into the abyss. But we've got to be careful not to scare off the new recruits. We're going to need them in the Great War when the machines take over.


----------



## mefizto (Mar 23, 2019)

Hi ralphbsz,

unlike linux->bsd, I do not find your posts "too discouraging".  This, of, course, is just another person's opinion, which I am just adding to the discussion so that you continue with your rigorous analysis and answers to the post(s) and not try to simplify the issue(s).

Kindest regards,

M


----------



## linux->bsd (Mar 26, 2019)

mefizto said:


> and not try to simplify the issue(s).



You're missing the point if you think the issue is about oversimplification. The discussion is about not letting the the shortcomings of the available solutions become a deterrence for using them.

The point is we should strive to present the shortcomings of available solutions, and the difficulties of implementing them, but then also describe the steps for implementing a workable solution that accomplishes the majority of the protection the requester is asking for. Really quite simple. The saying is "don't let the perfect be the enemy of the good."


----------



## mefizto (Mar 27, 2019)

Hi linux->bsd,



linux->bsd said:


> You're missing the point if you think the issue is about oversimplification.


That is, of course possible, in my experience, it is very difficult to expres oneself through written media.  To _wit_:  You have missed my point, which was that your's and mine are just opinions, and ralphbsz should take them as that.



linux->bsd said:


> The point is we should strive to present the shortcomings of available solutions, and the difficulties of implementing them, but then also describe the steps for implementing a workable solution that accomplishes the majority of the protection the requester is asking for.



We are in agreement, this is, what I was so in-artfully tried to express.

Kindest regards,
M


----------

