# HOWTO: Tuning L2ARC in ZFS



## t1066 (Feb 18, 2012)

General suggestions

1. Design the file system intelligently. Group frequently used files together.

2. Get enough cache capacity so that the working set fit into the cache.

3. Set secondarycache=all only for frequently used data. For example:

`# zfs set secondarycache=all data/frequent-used-data data`

Sysctl tunables

The following are all the sysctl tunables involving L2ARC and their defaults.


```
sysctl -a | grep l2arc

vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1
```

I will focus on the following 3 tunables.

1. vfz.zfs.l2arc_write_max

This tunable limits the maximum writing speed onto l2arc. The default is 8MB/s. So depending on the type of cache drives that the system used, it is desirable to increase this limit several times. But remember not to crank it too high to impact reading from the cache drives.

2. vfs.zfs.l2arc_write_boost

This tunable increases the above writing speed limit after system boot and before ARC has filled up. The default is also 8MB/s. During the above period, there should be no read request on L2ARC. This should also be increased depending on the system.

3. vfs.zfs.l2arc_noprefetch

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. To change it or not depends heavily on the workload of the system. On my system, the L2ARC cache hit ratio was less than 10% when it is set to the default value. After setting it to 0, the L2ARC cache hit ratio had now increased to 66%.

The following is what I had added to /etc/sysctl.conf.


```
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_write_max=26214400
vfs.zfs.l2arc_write_boost=52428800
```

The following is the *iostat* from my system when I write this howto:


```
iostat -xz -w 10 -c 10
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.5   2.7   585.1    73.2    0  18.1   3
da1        9.5   2.7   583.7    73.3    0  16.9   3
da3        9.4   2.7   584.0    73.3    0  17.3   3
da4        9.5   2.8   583.1    73.2    0  16.7   3
ada0      19.9   1.5  2375.9   157.5    0   1.5   1
ada1      19.4   1.6  2360.4   172.9    0   2.1   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.7   0.0   142.4     0.0    0  14.5   1
da1        2.3   0.0   104.6     0.0    0   5.1   0
da3        3.3   0.0   181.1     0.0    0  16.1   1
da4        2.4   0.0   129.2     0.0    0  10.7   1
ada0      28.9   0.2  3539.3    25.6    0   1.3   1
ada1      30.2   0.3  3506.9    38.4    0   1.5   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.8   0.0   115.2     0.0    0   8.2   1
da1        2.2   0.0   140.8     0.0    0  10.6   0
da3        1.8   0.0   115.2     0.0    0  11.5   0
da4        1.0   0.0    64.0     0.0    0   7.8   0
ada0      15.0   0.1  1841.4    12.8    0   1.4   1
ada1      16.7   1.2  2003.0   153.6    0   2.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        1.1   0.0    64.6     0.0    0   8.9   0
da1        1.4   0.0    89.6     0.0    0   9.8   1
da3        1.2   0.0    76.8     0.0    0  13.7   1
da4        1.8   0.0   115.2     0.0    0  14.4   1
ada0      14.3   0.1  1763.0    12.8    0   0.9   1
ada1      12.0   0.4  1311.9    51.2    0   0.9   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.6   5.5   169.8    29.8    0  10.2   2
da1        2.8   3.9   105.7    11.6    0   7.0   1
da3        2.6   3.9   154.2    11.6    0   5.5   1
da4        1.7   5.5    78.3    29.8    0   4.9   1
ada0      12.2   0.2  1368.7    14.4    0   1.0   0
ada1       9.2   0.9   874.0    65.6    0   0.9   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        5.2   0.0   297.2     0.0    0  13.7   2
da1        4.1   0.0   219.8     0.0    0  10.3   1
da3        4.0   0.0   243.8     0.0    0  23.5   2
da4        5.9   0.0   347.1     0.0    0  15.7   2
ada0      10.8   0.5  1279.1    52.8    0   1.0   0
ada1      13.0   0.1  1384.3     1.6    0   1.3   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        2.0   0.0    79.4     0.0    0  17.8   1
da1        2.3   0.0   104.5     0.0    0  20.0   1
da3        3.4   0.0   193.2     0.0    0  11.3   1
da4        3.5   0.0   205.7     0.0    0  18.8   2
ada0      18.2   0.1  2237.4    12.8    0   1.2   1
ada1      16.7   0.2  1801.8    25.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        4.8   5.9   259.5    27.8    0   6.5   2
da1        5.3   4.1   283.8    12.3    0  13.7   3
da3        3.8   4.1   243.2    12.3    0   7.7   1
da4        5.5   5.9   310.5    27.8    0   8.6   2
ada0      16.1   0.3  1912.6    27.2    0   1.3   1
ada1      14.0   0.4  1457.5    17.6    0   1.0   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        9.4   5.0   540.5    16.0    0  10.9   3
da1        8.6   3.5   513.9     8.7    0  13.1   3
da3        8.8   3.5   526.4     8.7    0  16.0   4
da4        9.9   5.0   579.0    16.0    0  14.1   3
ada0      30.1   1.0  3703.2   116.8    0   1.6   1
ada1      33.0   1.2  3921.3   108.8    0   1.6   1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        3.1   0.0   155.8     0.0    0   7.1   1
da1        3.5   0.0   205.7     0.0    0  13.5   1
da3        2.9   0.0   179.5     0.0    0  10.0   1
da4        3.5   0.0   193.5     0.0    0   8.3   1
ada0      12.8   0.7  1547.4    89.6    0   1.1   1
ada1      11.8   0.2  1301.5     3.2    0   1.4   1
```

da0, da1, da3 and da4 formed a ZFS pool, while ada0, ada1 are SSDs.

Reference

ZFS L2ARC (Brendan Gregg)
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c


----------



## Weaseal (Sep 14, 2012)

According to the wiki, you have these set incorrectly; /etc/sysctl.conf is not honored by ZFS:


> Note that this value is a run time sysctl that is read at pool import time, which means the default mechanism of setting sysctls in FreeBSD will not work. (Values set in /etc/sysctl.conf are set after ZFS pools are imported)



I believe these tunables should be moved to /boot/loader.conf


----------



## t1066 (Sep 17, 2012)

*Update information on sysctl tunables and add some examples*

The following are the sysctl tunables and their defaults for L2ARC. I have add a short description at the end of each tunables.


```
vfs.zfs.l2arc_write_max: 8388608    # Maximum number of bytes written to l2arc per feed
vfs.zfs.l2arc_write_boost: 8388608  # Mostly only relevant at the first few hours after boot
vfs.zfs.l2arc_headroom: 2           # Not sure
vfs.zfs.l2arc_feed_secs: 1          # l2arc feeding period
vfs.zfs.l2arc_feed_min_ms: 200      # minimum l2arc feeding period
vfs.zfs.l2arc_noprefetch: 1         # control whether streaming data is cached or not
vfs.zfs.l2arc_feed_again: 1         # control whether feed_min_ms is used or not
vfs.zfs.l2arc_norw: 1               # no read and write at the same time
```

As mention in the above post, changes to vfs.zfs.l2arc_write_max will not affect the behaviour of a running pool. For best compatibility, it should be set in /boot/loader.conf.

The other interesting tunables, vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again do not have such restrictions. You can set them in the command line, and it will immediately take effect.

If vfs.zfs.l2arc_feed_again=1, then when the data that are going to written onto L2ARC is larger than vfs.zfs.l2arc_write_max, the feeding period of L2ARC is reduced, up to vfs.zfs.l2arc_feed_min_ms. To illustrate how it works, here are three ways to set maximum write speed on L2ARC to 80MB/s


```
vfs.zfs.l2arc_write_max=83886080
vfs.zfs.l2arc_feed_again=0
# feed 80MB per second
```

or


```
vfs.zfs.l2arc_feed_min_ms=100
# feed 8MB per 100 ms
```

or


```
vfs.zfs.l2arc_write_max=16777216
# feed 16MG per 200 ms
```

When the L2ARC consists only one drive, the difference is not much, but if the L2ARC consists 2 drives, cache1 and cache2, then in the above example, when L2ARC is writing at its maximum speed, we will see


```
80MB written to cache1 in the first second
80MB written to cache2 in the next second
```

or


```
8MB written to cache1 in the first 100ms,
8MB written to cache2 in the next 100ms, etc.
```

or


```
16MB written to cache1 in the first 200ms,
16MB written to cache2 in the next 200ms, etc.
```


----------



## gkontos (Sep 17, 2012)

I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.

That would only be needed if they can not be adjusted without a reboot.


----------



## t1066 (Sep 18, 2012)

Actually, you may adjust vfs.zfs.l2arc_write_max on the command line also. But it will not take effect until you export and then import back your pool. If the data pool is not separately from the root, this have to set on /boot/loader.conf instead of /etc/sysctl.conf. And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.

However, you can change vfs.zfs.l2arc_feed_min_ms, vfs.zfs.l2arc_noprefetch, vfs.zfs.l2arc_feed_again on the command line and it will immediately change the behviour of the pool without a need for export/import.

vfs.zfs.l2arc_feed_secs and vfs.zfs.l2arc_norw seem to have reasonable defaults.


----------



## arabesc (Sep 14, 2015)

Weaseal said:


> I believe these tunables should be moved to /boot/loader.conf





gkontos said:


> I think he means that those variables should go to /boot/loader.conf instead of /etc/sysctl.conf.





t1066 said:


> And it is also better to put vfs.zfs.l2arc_write_boost on /boot/loader.conf.


I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
My system version is 10.2.


----------



## borov (Dec 21, 2015)

arabesc said:


> I have placed those modified vfs.zfs.l2arc_write_max and vfs.zfs.l2arc_write_boost variables to /boot/loader.conf, but they have default values after system reboot. What's wrong?
> My system version is 10.2.


I have similar problem.

Also this setting do not work.
`sysctl vfs.zfs.l2arc_noprefetch="0"`


----------



## t1066 (Dec 22, 2015)

If you can change the values in the command line, they should be to put in /etc/sysctl.conf.


----------



## grahamperrin@ (Nov 22, 2021)

borov said:


> … this setting do not work.
> 
> `sysctl vfs.zfs.l2arc_noprefetch="0"`



I set it for myself around an hour ago (without using quotation marks),

`sysctl vfs.zfs.l2arc.noprefetch=0`

Also set in /etc/sysctl.conf.

How can I tell whether the setting is effective?

Partly related: 









						tuning(7) – FreeBSD performance tuning manual
					

Before I open a bug for a part of the page that's wrong:   does anyone else have corrections, or suggestions for improvement?  Thanks




					forums.freebsd.org


----------



## Alain De Vos (Nov 22, 2021)

Google showed me this,


			Chris's Wiki :: blog/solaris/ZFSL2ARCNoprefetchTunable
		

Don't touch it unless you really know what your doing ?


----------



## grahamperrin@ (Nov 23, 2021)

Thanks, no observable downside yet. I'll observe things over the next few days or weeks.


  

– second and third shots taken after using the File menu of Recoll deskutils/recoll to update its index. (I also run the daemon for real-time indexing of changed files.)

Cache devices for ada0 are currently at da0 and da1.


```
% zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Nov 23 05:42:34 2021
------------------------------------------------------------------------

L2 ARC Summary: (DEGRADED)
        Low Memory Aborts:                      1.97    k
        Free on Write:                          14.70   k
        R/W Clashes:                            20
        Bad Checksums:                          185.12  k
        IO Errors:                              0

L2 ARC Size: (Adaptive)                         57.91   GiB
        Decompressed Data Size:                 137.19  GiB
        Compression Factor:                     2.37
        Header Size:                    0.13%   184.88  MiB

L2 ARC Evicts:
        Lock Retries:                           8
        Upon Reading:                           0

L2 ARC Breakdown:                               3.82    m
        Hit Ratio:                      65.29%  2.50    m
        Miss Ratio:                     34.71%  1.33    m
        Feeds:                                  85.53   k

L2 ARC Writes:
        Writes Sent:                    100.00% 17.05   k

------------------------------------------------------------------------
```


----------



## t1066 (Nov 23, 2021)

The change in sysctl should take effect immediately.

Since each case is different, the best way to determine whether a setting is good or bad is to try it out.


----------



## grahamperrin@ (Nov 25, 2021)

From zfs.4 — OpenZFS documentation:


*l2arc_noprefetch=1|0 (int)*

Do not write buffers to L2ARC if they were prefetched but not used by applications. In case there are prefetched buffers in L2ARC and this option is later set, we do not read the prefetched buffers from L2ARC. Unsetting this option is useful for caching sequential reads from the disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial in case the L2ARC device is significantly faster in sequential reads than the disks of the pool.

Use *1* to disable and *0* to enable caching/reading prefetches to/from L2ARC.

Postscript

After four weeks of caching/reading prefetches to/from L2ARC:

qualitatively, I haven't become aware of any problem
measurably, <https://forums.freebsd.org/posts/547868> looks more than good enough to me
– so I have no plans to revert from `0` to (the default) `1`.


----------

