# Creating a mirror of encrypted swap partition



## stratacast1 (Mar 17, 2019)

I'm putting a new drive in my server so I can mirror my drive that has freebsd-boot, swap, and freebsd's zroot. The boot and zroot parts are straightforward, but I'm not too sure about how to handle swap. I set encrypted swap on when I setup the system so to my understanding, it's using GELI

```
/dev/ada0p2.eli         none    swap    sw              0       0
```

My newly created partition is `/dev/ada3p2`. How should I mirror these? Is this going to be the appropriate way here? Thread howto-convert-single-disk-zfs-on-root-to-mirror.49702/

Or here? I would think with this solution, the swap partitions wouldn't be mirrored.


----------



## D-FENS (Mar 17, 2019)

Creating mirrored swap is quite easy, see below. However, I changed my mind about mirroring swaps. I think mirrored swap does not bring any advantage - why keep multiple copies of the same data in the swap?
Normally mirroring is used to ensure redundancy, but data in swap is normally not that critical so even if the disk fails and the swap is unavailable - a program may crash but after a restart everything is fine.
It's probably better to just add the swap partitions separately and provide a bigger swap space.

This is how I did mirrored encrypted swap (the important line is in the fstab):

```
# Optional: Configure swap mirror on the HDDs (if you have a spare partition ~ 32GB, or an empty disk)
# DESTROYS DATA ON HDD:
#gpart destroy -F ada0
#gpart create -s GPT /dev/ada0
gpart add -t freebsd-swap -s 32G -a 1M -l hddswap0 /dev/ada0
gpart show ada0
# (repeat for all HDDs)
# Create mirror of 4 HDDs:
gmirror load
gmirror create hddswap gpt/hddswap0 gpt/hddswap1 gpt/hddswap2 gpt/hddswap3
gmirror status
# Set automatic encrypted SWAP creation on boot:
vi /etc/fstab
#---------------------------8<------------------------------
# Device                Mountpoint      FStype  Options         Dump    Pass#
# Encrypted swap
/dev/mirror/hddswap.eli none swap sw,ealgo=AES-XTS,keylen=128,sectorsize=4096 0 0
#---------------------------8<------------------------------
swapon -a
swapinfo
```


----------



## D-FENS (Mar 17, 2019)

P.S. In general it is not necessary to use separate encryption keys and separate encryptions for each swap partition. So in my example all are mirrored first and then I put the encryption on top of the mirror.


----------



## stratacast1 (Mar 18, 2019)

I see your point for not needing to worry about losing data in swap, but that crash is a risk in availability. Would FreeBSD auto restart those processes? I like the idea of mirrored swap because if a drive dies it will just continue without a crash


----------



## ralphbsz (Mar 18, 2019)

How often will a drive die (or have serious enough read errors that a process will crash)?  Very rarely.
What fraction of all the processes on your computer would cause a crash or require a reboot if they stopped?  Many will not.  And swap isn't even used much on most systems, so even if the swap disk dies, most processes will live.
How often do you reboot your computer voluntarily?  Depends; my laptop and rack-mounted servers roughly weekly, my home server roughly monthly (at home due to power outages), YMMV.
How often does your computer crash involuntarily?  Mine hardly ever.  My FreeBSD server crashes perhaps every year or two.
Put in reasonable estimates, and you will find that the improvement in availability from mirrored swap is small.

Now on the other side, what is the cost of mirrored swap, if you already have two disks?  A little more disk space, irrelevant for most real-world uses (swap is tiny on the scale of modern disks).  A tiny bit of work to set up.  Tiny little loss of performance (need to wait for two writes that are occurring in parallel, which is not much slower than waiting for one of them).  Compare that tiny or non-existing cost to the small benefit, and it is probably still a win.  So go do it if you want.

On the question of auto restart: For the most part, no.  The System-V init system (used by many Linux distributions before systemd) has the option to auto-restart services that fail.  This was initially intended for getty, the program that gets the username from a terminal and passes it to login, and needs to be restarted after the user logs out.  The BSD-style rc system does not in general restart a service after it crashes.  There may be daemons that restart some subprocesses (I have one of those at home, but I wrote that code myself).

Anecdote from long ago: In a job a long time ago (when dinosaurs roamed the earth), we used an early 4-processor Unix machine for our departmental server, for a computer science research group with roughly a dozen people.  One time we noticed that nobody had received any mail, for over a day.  A quick look at the mail log found that no mail message had entered or left our server (and therefore our group) for a day and a half.  This tells you that this was at a time when e-mail wasn't that important yet (today I would notice in roughly 15 minutes).  So we looked at the server logs, and discovered that the OS had detected the complete failure of one of the four CPUs, and for safety shut that CPU down.  The process that was running on that CPU was crashed, and it happened to be sendmail, which explains the absence of mail.  But the rest of the computer continued functioning so normally that none of us had even noticed!  I guess we weren't very performance constrained, and nobody happened to look at the blinking yellow light on the front panel.  We simply restarted sendmail, and kept running on a 3-processor machine.  In the meantime we contacted field service, who told us that a replacement CPU is a "customer replaceable part", and they mailed us a new CPU.  Which arrived via courier service about 24 hours later.  In those days, a "CPU" was a whole PC board, about the size of an A4 sheet of paper.  The amazing thing was the instructions: it was actually not necessary to turn the computer off or even reboot it to take the broken CPU out and put the spare part in; it could be hot-swapped, and then enabled via a command.  We decided that we were to chicken to try that, shut the machine down, put the spare in, and rebooted.  How is that for availability and uptime?

Same site, same machine, about a year earlier: When I interviewed there to get the job, the manager of the group gave me a little tour of the computer room, including the group's main server (see above).  I had one of those new-fangled RAID arrays with many disks (probably 10 or 20, which was considered a heck of a lot in those days).  And to show it off, he pulled a disk physically out of the disk array, handed it to me still warm, I handed it back (a little astonished), he plugged it in, and everything continued functioning.  In his defense, the disk was not pulled from the production array connected to the main server, but from a second RAID array used on an experimental system.  Still, having the guts to pull a disk from a running system amazed me.

This was about 20 or 25 years ago.  Back then, they already had machines with really good OSes and availability.


----------



## D-FENS (Mar 18, 2019)

stratacast1 said:


> I see your point for not needing to worry about losing data in swap, but that crash is a risk in availability. Would FreeBSD auto restart those processes? I like the idea of mirrored swap because if a drive dies it will just continue without a crash


If you need high availability - sure, go for it. I completely agree. It's just that in such server scenarios it's probably impractical to use swap anyway, because of performance etc. The server's RAM should be calculated so that swap is not needed.

In a NAS or workstation scenario when a drive dies I see the crash actually as a good thing. It would be like an alarm bell before any other drives fail. But there is no silver bullet, for sure. Use mirrored swap if your use case would benefit from it. I just don't see one for myself.

I forgot to mention, mirroring would make of course swap read times better. This might be a plus.


----------



## D-FENS (Mar 18, 2019)

stratacast1 said:


> I see your point for not needing to worry about losing data in swap, but that crash is a risk in availability. Would FreeBSD auto restart those processes? I like the idea of mirrored swap because if a drive dies it will just continue without a crash


By the way, you could also go for a compromise. For example if you have 4 drives, instead of mirroring them, you could create a RAIDZ1 encrypted swap and have 3 times the size, while still tollerating single drive failure. Or you could use 1/3 of the size on each partition and then have the same swap size while saving up on storage.


----------



## D-FENS (Mar 18, 2019)

ralphbsz said:


> The amazing thing was the instructions: it was actually not necessary to turn the computer off or even reboot it to take the broken CPU out and put the spare part in; it could be hot-swapped, and then enabled via a command.  We decided that we were to chicken to try that, shut the machine down, put the spare in, and rebooted.  How is that for availability and uptime?


WOW! Now I have even more respect for the ancients!


----------



## stratacast1 (Mar 20, 2019)

roccobaroccoSC said:


> By the way, you could also go for a compromise. For example if you have 4 drives, instead of mirroring them, you could create a RAIDZ1 encrypted swap and have 3 times the size, while still tollerating single drive failure. Or you could use 1/3 of the size on each partition and then have the same swap size while saving up on storage.



I think I'll go with the mirror because the small writes are probably negligible in the greater performance of the system, and the added bonus of quicker reads might be to my benefit. My swap is barely ever used though to be completely honest, but it's just nice to know that drive a == b in every respect. Plus I'll be going away for a month and the likelihood of error is so small, but it would be a shame to have swap cause issues while I'm away. I only have 2 drives for this right now so I can't do any fancy RAIDZ1 stuff (as much as I really want to) but this really helps


----------



## D-FENS (Mar 20, 2019)

stratacast1 said:


> I think I'll go with the mirror because the small writes are probably negligible in the greater performance of the system, and the added bonus of quicker reads might be to my benefit. My swap is barely ever used though to be completely honest, but it's just nice to know that drive a == b in every respect. Plus I'll be going away for a month and the likelihood of error is so small, but it would be a shame to have swap cause issues while I'm away. I only have 2 drives for this right now so I can't do any fancy RAIDZ1 stuff (as much as I really want to) but this really helps


Well, if you have only two drives and need redundancy, I think the decision already took itself, right?


----------

