# ZFS SSD Tuning for a Desktop Use



## freezr (Jan 14, 2022)

Dear All,

looking for some info to tuning and extend the life of my SSD disks I couldn't find any valuable information.

The documentation reports:



> _vfs.zfs.vdev.trim_on_init_ - Control whether new devices added to the pool have the TRIM command run on them. This ensures the best performance and longevity for SSDs, but takes extra time. If the device has already been secure erased, disabling this setting will make the addition of the new device faster. Adjust this value at any time with sysctl(8).





> _vfs.zfs.l2arc_write_max_ - Limit the amount of data written to the L2ARC per second. This tunable extends the longevity of SSDs by limiting the amount of data written to the device. Adjust this value at any time with sysctl(8).





> _vfs.zfs.l2arc_write_boost_ - Adds the value of this tunable to vfs.zfs.l2arc_write_max and increases the write speed to the SSD until evicting the first block from the L2ARC. This "Turbo Warmup Phase" reduces the performance loss from an empty L2ARC after a reboot. Adjust this value at any time with sysctl(8).





> L2ARC is the second level of the ZFS caching system. RAM stores the primary ARC. Since the amount of available RAM is often limited, ZFS can also use cache vdevs. Solid State Disks (SSDs) are often used as these cache devices due to their higher speed and lower latency compared to traditional spinning disks. L2ARC is entirely optional, but having one will increase read speeds for cached files on the SSD instead of having to read from the regular disks. L2ARC can also speed up deduplication because a deduplication table (DDT) that does not fit in RAM but does fit in the L2ARC will be much faster than a DDT that must read from disk. Limits on the data rate added to the cache devices prevents prematurely wearing out SSDs with extra writes. Until the cache is full (the first block evicted to make room), writes to the L2ARC limit to the sum of the write limit and the boost limit, and afterwards limit to the write limit. A pair of sysctl(8) values control these rate limits. vfs.zfs.l2arc_write_max controls the number of bytes written to the cache per second, while vfs.zfs.l2arc_write_boost adds to this limit during the "Turbo Warmup Phase" (Write Boost).



None of these make sense for me.

My knowledge of FreeBSD is still very limited and my knowledge of ZFS is even narrower.
On Linux I can enable fstrim through a cron task or systemd, I can also reduce swappiness and enable some other options on the fstab file, and this would be enough for a desktop use (in Linux).

So far I don't have my computer with me, but if my description is enough, maybe someone can already give me some advices.

I installed FreeBSD with ZFS-ON-ROOT over two 250 GB SSD in stripe mode, I already noticed that there is anything inside my fstab file hence the disks are handled somewhere else by ZFS. Maybe I heard from Allan Jude in a episode of BSDNow that ZFS is smart enough to handle SSD maintenance by itself, but I am not very sure.

I heard and read that you can tune ZFS for particular cases but mine user case is mainly a desktop use: web browsing, emailimg, compiling from time to time, graphic design, sometimes Blender and video editing.

Thanks in advance, I can share more info about my disks later in the evening.

tgl


----------



## Menelkir (Jan 14, 2022)

ZFS have his own method of trim 
	
	



```
zpool trim pool
```
if you use fstrim, I guess you're doing two trims.
I usually have my swap on rotational disks, since swap isn't something that will be used often, but YMMV.


----------



## grahamperrin@ (Jan 14, 2022)

If you don't use a hard disk drive, you can probably ignore L2ARC. 

I do use L2ARC, *love* it. 

OpenZFS: L2ARC: CACHE vdev devices : freebsd



tgl said:


> ZFS is smart enough to handle SSD maintenance



<https://openzfs.github.io/openzfs-docs/man/8/zpool-trim.8.html>


----------



## Beastie7 (Jan 14, 2022)

It might help with explaining exactly what you’re trying to accomplish with this task. ZFS is huge.


----------



## Erichans (Jan 14, 2022)

ZFS has a lot of knobs. Perhaps these will get you started:

 ZFS 101—Understanding ZFS storage and performance
 ZFS for Newbies by Dan Langille
 Today's ZFS Michael W Lucas
Important when you start defining your pool layout: click. Depending on the chosen layout configuration, changes and disk additions are limited.

For further in depth and reference the two readable books about using ZFS: FreeBSD Development: Books, Papers, Slides.

At various places on this forum, there are other links but, as mentioned above: your use case matters!

Very short answer on some topics: don't use deduplication. If you have an SSD, or better NVME SSD, or even spinning rust platters; _and_ a decent amount of ram (say > 8GB) for your system and the ZFS ARC then, most likely you don't need a L2ARC; more:

 OpenZFS: All about the cache vdev or L2ARC - Klara Systems - June 26, 2020
___
Edit: videos #2 & #3 are both for beginning ZFS users, both cover quite a lot (ZFS has a lot to offer). I mainly gave these two so you have a choice; I think that either one will give you a worthwhile introduction of the basics of ZFS.


----------



## grahamperrin@ (Jan 14, 2022)

tgl said:


> The documentation reports:



It's partly outdated. 



tgl said:


> My knowledge of FreeBSD is still very limited and my knowledge of ZFS is even narrower.



You need not have deep knowledge of ZFS for desktop use. Keep it simple  

Scrub, and backup.


----------



## freezr (Jan 14, 2022)

Beastie7 said:


> It might help with explaining exactly what you’re trying to accomplish with this task. ZFS is huge.



Anything special, check below:


> mine user case is mainly a desktop use: web browsing, emailimg, compiling from time to time, graphic design, sometimes Blender and video editing.


----------



## freezr (Jan 14, 2022)

Menelkir said:


> ZFS have his own method of trim
> 
> 
> 
> ...


Is this something to run weekly? I use a swapfile, is it a good recommendation reduce swappiness since I have 16GB or ram? 

Thanks!


----------



## freezr (Jan 14, 2022)

I'd like to clarify that I installed ZFS-ON-ROOT through the automatic wizard provided by the installer, the only configuration I made was to stripe the two disks to have 500GB of storage.


----------



## grahamperrin@ (Jan 14, 2022)

FreeBSD bug 261212 – Update the ZFS chapter (20) of the FreeBSD Handbook


----------



## Erichans (Jan 14, 2022)

tgl said:


> I'd like to clarify that I installed ZFS-ON-ROOT through the automatic wizard provided by the installer, the only configuration I made was to stripe the two disks to have 500GB of storage.



I missed that, if you indeed have a stripe over the two 250GB disks and not have them in a mirror then, I strongly advise you to reconsider now: you have no redundancy. If there is an error on one disk you lose all your data!

Look into the layout configuration options and their redundancy. It is like a traditional RAID0 layout and with that one time referring to traditional RAID, I immediately say that, when thinking about ZFS, you have to let go of thinking in traditional RAID terminology. If you want a quick start-over with two 250GB SSDs: make a mirror (and have a look at the article or the ZFS youtube videos )


----------



## mer (Jan 14, 2022)

I'll second what Erichans says here.  If we're understanding how you have it configured (the output of zpool status will tell us exactly) any problem in either device will result in a non bootable system.  Mirrors are good and yes you only have "disk size" space.

As for your very original post, the defaults work very nicely for a desktop system.  Don't worry about L2ARC (it's a secondary cache and can make a difference on a server with specific workloads), trim the defaults are fine.


----------



## Menelkir (Jan 14, 2022)

tgl said:


> Is this something to run weekly? I use a swapfile, is it a good recommendation reduce swappiness since I have 16GB or ram?
> 
> Thanks!


Don't use a swapfile at all. Use a swap partition, even if it's a 500Mb partition, it will be better than a swapfile.
I have 16Gb of ram too. I have two HDD (that's a pool called zdata, my zroot is on single SSD), and two swap partitions of 2G on each HDD.


----------



## zirias@ (Jan 14, 2022)

Just adding: A swap file on ZFS can even be dangerous (cause deadlocks) because ZFS needs to dynamically allocate memory itself. So, under heavy memory pressure, writing this one inactive page to swap that's so desperately needed to free a page of physical RAM could fail because ZFS can't get the RAM it needs to do this...

Yes, always use swap partitions.


----------



## freezr (Jan 15, 2022)

```
$ zfs list
NAME                   USED  AVAIL     REFER  MOUNTPOINT
zroot                  199G   241G       96K  /zroot
zroot/ROOT            4.78G   241G       96K  none
zroot/ROOT/default    4.78G   241G     4.16G  /
zroot/ROOT/pre220107     8K   241G     3.87G  /
zroot/tmp              172K   241G      172K  /tmp
zroot/usr              194G   241G       96K  /usr
zroot/usr/home         193G   241G      193G  /usr/home
zroot/usr/ports        786M   241G      786M  /usr/ports
zroot/usr/src          702M   241G      702M  /usr/src
zroot/var             1.10M   241G       96K  /var
zroot/var/audit         96K   241G       96K  /var/audit
zroot/var/crash         96K   241G       96K  /var/crash
zroot/var/log          580K   241G      580K  /var/log
zroot/var/mail         160K   241G      160K  /var/mail
zroot/var/tmp           96K   241G       96K  /var/tmp
```


```
$geom disk list
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 240057409536 (224G)
   Sectorsize: 512
   Mode: r2w2e4
   descr: KINGSTON SA400S37240G
   lunid: 50026b7682359175
   ident: 50026B7682359175
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: Samsung SSD 860 EVO 250GB
   lunid: 5002538e40f0f259
   ident: S3YHNX0M415404N
   rotationrate: 0
   fwsectors: 63
   fwheads: 16
```

I am not concerned using striped disks, I have been always using it without issues (lvm/ext4, btrfs) so far, I understand the risks but for me is fine.

I heard that Allan Jude suggested to use a physical partition for ZFS, however I allocated 4GB for the swap file and I hope this shouldn't represent a risk if I also reduce the access to swap memory. I may try to reduce the Linux partition to recover 4GB if really having a swap file is considered a danger. 

By the way the latter topic isn't specified anywhere in the handbook, I personally consider the swap partition something old and space wasted, however I found a suggestion this very forum that could help me out:


```
# Prefer losing cache over swapping
vm.defer_swapspace_pageouts=1
```









						ZFS - SWAP partition or swap file?
					

Greetings all,  I am planning new server installation.  The operating system will reside on SSDs (mirrored) and the data on mechanical hard drives.  I am wondering whether to use SWAP partition or swap file.  As best as I understand, a SWAP partition needs to be mirrored and only the allocated...




					forums.FreeBSD.org


----------



## grahamperrin@ (Jan 15, 2022)

mefizto said:


> … What would be the best practice for a _stable _system? …





lkateley said:


> … Maybe someone who knows more about what might be swapping/paging on a freebsdFreeBSD system will have more insight, …





tgl said:


> … I heard that Allan Jude suggested to use a physical partition for ZFS, however I allocated 4GB for the swap file and I hope this shouldn't represent a risk if I also reduce the access to swap memory. …





mark_j said:


> … this bug.
> 
> In reality, swap on ZFS is no more problematic than mounting anything else. …





grahamperrin said:


> Swap deadlock in 0.7.9 · Issue #7734 · openzfs/zfs
> 
> 
> System information Type Version/Name Distribution Name Debian Distribution Version stretch Linux Kernel 4.16.0-0.bpo.2-amd64 Architecture amd64 ZFS Version 0.7.9-3~bpo9+1 SPL Version 0.7.9-3~bpo9+1...
> ...



2021-10-10:



> … I found that the issue is still not solved.


----------



## Lamia (Jan 15, 2022)

You can leave swap management for pkgs like swapd and swapmon in freshports. 
You could alternatively use zvol for swap and adjust its size over time according to your need.


----------



## thindil (Jan 15, 2022)

If you plan to do some tasks which require a lot of memory and data read from disc, like for example playing in some AAA modern games (from my experience, yes it is possible on FreeBSD), I would recommend to set maximum memory size for ZFS cache, ARC. I was a few times in a situation, when a game crashed due to lack of RAM, caused by the default setting of ZFS to use 5/8 of all RAM. Most guides recommend setting it 1 GB RAM per 1 TB of disk. In my case, setting it to double of this value not only prevented crashes mentioned above, but I also noticed that the whole system started works a bit faster. Mostly because it stopped trying to load to ARC all these games' data and more often hit the cache. But, as they say, YMMV.


----------



## Crivens (Jan 15, 2022)

Lamia said:


> You can leave swap management for pkgs like swapd and swapmon in freshports.
> You could alternatively use zvol for swap and adjust its size over time according to your need.


Beware where that zvol is located. It will be completely duplicated for each snapshot you do. So anything that does snapshots regulary might become a disk space hog.


----------



## grahamperrin@ (Jan 15, 2022)

Lamia said:


> alternatively use zvol for swap



Ignoring the risk? 

I mean, it _might_ be deadlock-free for many people most of the time, however: with users (of OpenZFS) still reporting deadlocks, I shouldn't recommend it (as an alternative) to someone who's finding their feet. 

From <https://openzfs.github.io/openzfs-docs/man/8/zfs-create.8.html#ZFS_Volumes_as_Swap>:



> … ZFS volumes may be used as swap devices. …



– however I don't expect manual pages such as this to list bugs such as OpenZFS issue 7734.


----------



## Lamia (Jan 15, 2022)

Crivens said:


> It will be completely duplicated for each snapshot you do.


Turn snapshot creation off for the zvol.


----------



## Lamia (Jan 15, 2022)

All of these depend on the workload, the default swap size "2gb" on fresh installation most times suffice. For a desktop with spinning rust, get an SSD into the SSD cache SATA slot on the motherboard (some boards have it) and use it for layer 2 ARC, you will notice a performance boost.

 The need for cache (and log) comes at a time when the machine has limited resources (RAM mostly) yet gets heavy workload thrown at it.


----------



## freezr (Jan 15, 2022)

The documentation about making a swap on a zvol is really scarce for FreeBSD:






						FAQ — OpenZFS  documentation
					






					openzfs.github.io
				




I think I had better to shrink the Linux partition and recover 4 or at least 2GB... 

By the way this has been deprecated in FreeBSD 11:


```
# Prefer losing cache over swapping
vm.defer_swapspace_pageouts=1
```


----------



## grahamperrin@ (Jan 15, 2022)

tgl said:


> deprecated



Thanks, I removed the quote from my earlier comment.

Confirming that there's no mention in FreeBSD documentation:


```
root@mowa219-gjp4-8570p-freebsd:~ # gh repo sync grahamperrin/freebsd-doc && cd /usr/doc && gh repo sync && git -C /usr/ports pull --ff-only && git -C /usr/src pull --ff-only && cd
✓ Synced the "grahamperrin:main" branch from "freebsd:main"
✓ Synced the "main" branch from grahamperrin/freebsd-doc to local repository
Already up to date.
Already up to date.
root@mowa219-gjp4-8570p-freebsd:~ # logout
% time grep -r vm.defer_swapspace_pageouts /usr/doc/documentation/
0.137u 0.420s 0:10.66 5.1%      26+180k 4406+0io 0pf+0w
%
```


----------



## vermaden (Jan 15, 2022)

There are two ZFS tunables I know that may be useful for FreeBSD on the Desktop.



1. Limit the number of TRIM instructions sent on burst to device.

By default 64 TRIM requests are sent to SSD device. I once had SSD device that took 500ms (half a second) for each TRIM operation) so on the default 64 TRIM instructions the system almost frozen when I deleted large files. I set that to 1 and system was smooth as usual. The only downside is that TRIM instructions will be issued later, but that does not change anything really.

*/etc/sysctl.conf*

  vfs.zfs.vdev.trim_max_active=1



2. Really force small ARC.

... and by really I mean REALLY. The vfs.zfs.arc.min and vfs.zfs.arc.max are not always respected ... but there is also vfs.zfs.arc_free_target which can be used to REALLY force ARC to be small. The tricky part is that you need to find your value.

For *32 GB* of RAM its *1024 x 1024 x 7.75* which means *8126464* value.

For *16 GB* RAM its *1024 x 1024 x 3.85* which means *4037017* value.

Example setting in */etc/sysctl.conf *for 16 GB RAM:

  vfs.zfs.arc_free_target=4037017

Keep in mind that these are 'maximum' values which mean 'smallest' ARC size.



Other things.

Disable *prefetch*.

Disable ATIME with *atime=off* for entire ZFS pool.



Hope that helps.


----------



## grahamperrin@ (Jan 15, 2022)

vermaden said:


> also `vfs.zfs.arc_free_target`





There's another, I don't claim to have any experience with it:

<https://old.reddit.com/r/freebsd/comments/pvsu2w/-/hecksww/?context=1>



> `vfs.zfs.arc.sys_free`


----------

