# Mirror Swap Or Not ?



## Geezer (Jun 5, 2021)

When doing a new install with zfs on 2 mirrored nvmes, what are the advantages/disadvantages of mirroring swap?

Thank you.


----------



## im (Jun 5, 2021)

I predict a system failure like  'kernel panic' in case of failing unmirrored swap device.
So I prefer to have a complete mirror of all necessary data partitions.


----------



## zirias@ (Jun 5, 2021)

IMHO: Don't.

You want swap to be as fast as possible. Define a swap space on every available physical drive, the OS will make sure to spread the load equally. In the best case, swapping in/out of several pages can happen in parallel on all the drives.

Mirroring would help to keep data in swap safe from bitrot or even a failing drive. But the worst thing that could ever happen would be a crash of the system. This is an unlikely event, and the price you pay is slower swapping. Swapping (especially when the drives aren't ultra-fast) can greatly impair the performance of the whole system, so IMHO, don't pay that price…


----------



## im (Jun 5, 2021)

If you care about performance and your machine is swapping too much then just increase the amount of RAM or replace machine with the better one instead of speeding swapping.


----------



## zirias@ (Jun 5, 2021)

Nope. Swapping should be an edge-case, of course. But _when_ it's happening, you don't want your whole machine to stall to death. Swapped pages are transient data and, ideally, short-lived.


----------



## asteriskRoss (Jun 7, 2021)

I'm with Zirias. Swap is essentially an extension of RAM and I certainly don't mirror my RAM.


----------



## gpw928 (Jun 8, 2021)

I think that the answer depends on your ability to tolerate an outage.

If the down time is of no great consequence, then you can focus on performance.

Some systems require maximum up-time.  Such systems require redundancy built in at every level (including swap).

Redundant storage is going to cost some performance, but you can mitigate.  e.g. RAID-10 will spread the load to multiple spindles, and also reduce seek time for reading.  But there will be some penalty for writing.

If (reliable) performance really matters move your swap to (mirror'd) NVMe and/or get more (ECC) main memory.


----------



## Alain De Vos (Jun 8, 2021)

If you can't trust swap , is like not be able to trust your ram.
Maybe raid swap if you use special hardware ...


----------



## zirias@ (Jun 8, 2021)

Bottom line is: _you don't want your system to swap at all_. A system that constantly needs swap has too little RAM for its workload. Swap is designed to mitigate rare and temporary situations of memory pressure without having to kill processes immediately.

The major problem with swapping has always been the abysmal performance. Of course, this is _much_ better on modern NVMe drives (to the extent of providing acceptable performance depending on the workload), but even with them, there's a remarkable slowdown compared to direct RAM access. This is also the reason why having too much swap might be bad. In a situation when every memory access will trigger a page fault, the system might become too slow to be usable at all, so the OOM killer suddenly looks nicer 

Now, imagine you were running out of memory without any swap: The OOM killer would have to pick some process to kill, so your system will be unstable. Swap avoids that. Do you really want to slow that down, just to protect yourself from the unlikely case of some error that could corrupt the temporarily swapped pages? Even pages in physical RAM _can_ become corrupted. ECC RAM is a way to (somewhat) protect against that.

There's also a technical reason to avoid any additional software layers below your swap areas: If this code has to dynamically allocate memory, you could run into a deadlock (halting the whole system immediately) when there's too little physical RAM available to satisfy that allocation. I learned this can even happen with GELI, so on my server, I disabled GELI for swap after running into a deadlock (for a _very_ few times). It's discouraged to place swap on top of ZFS for such reasons.

---
Disclaimer: There are very specific workloads for which what I wrote isn't true. One example is a heavy multiuser system with many interactive tasks (of different users) that are idle most of the time. For such a system, it makes sense to use a lot of swap for storing the working set of the idle tasks. Another example would be running an application that is itself optimized for swap usage.


----------



## mer (Jun 8, 2021)

Another vote for the logic behind Zirias replies.  
If one has enough memory, the only other use I can think of for swap is kernel core files (at least that was the default location for them at one time).
I don't mind having swap partitions on multiple devices, it's been a long time since I've seen any of it used and given the size of "small" disks (I'm including NVME, SSDs and spinning platters here) a 2GB partition is almost noise (I'm excluding embedded systems, that is different ballgame).

So partition the two devices the same assuming these are a bootable pair of devices:
boot partition (either freebsd-boot or efi)
swap (2GB)
filesystem (freebsd-zfs)

The boot and swap partitions don't get mirrored, the filesystem one does.  Why multiple boot partitions?  Because the bios will find either one of them and be able to boot.  By the time you hit single user mode the mirror should be up and running.
don't forget to align the partitions at appropriate block sized boundaries (typically if you say align at 1M boundaries you're good because it is inherently a multiple of 512 and 4096).


----------

