# Playing with ZIL/L2ARC



## bbzz (May 26, 2012)

Hey everyone,

I managed to get one extra SSD to play around with as cache/log device on my home server (3x2 mirror vdevs), and I tried a bunch of different configurations. My understanding of how this all works is pretty much limited to what I know from forums. So I ran a couple of tests and have a few questions, maybe you could help.

All my disks are encrypted, so at first I assumed that I would need to encrypt both ZIL and L2ARC to keep it fully encrypted again. So, for ZIL, I figured, that makes sense.

But what about L2ARC? From my understanding of how this works, L2ARC must be pointed from main memory. So, after a reboot, I would assume that contents of L2ARC can't be retrieved. How would it be possible to retrieve this data, and so, is encrypting this L2ARC necessary?

As it turns out, however, *bonnie++* tests showed pretty much same and consistent results whether I used log/cache or not. For example this is with both log/cache on same SSD:

```
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
zhenbox.home.do 96G   142  99 68457  13 44808  11   366  96 144211  13 180.3   3
Latency               132ms    6085ms    5642ms     117ms     386ms     741ms
Version  1.96       ------Sequential Create------ --------Random Create--------
zhenbox.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 27137  97 +++++ +++ 27790  98 21062  97 +++++ +++ 12205  99
Latency             12448us      97us     167us   25335us      82us     237us
1.96,1.96,zhenbox.home.domain,1,1337982754,96G,,142,99,68457,13,44808,11,366,96,144211,13,180.3,3,16
,,,,,27137,97,+++++,+++,27790,98,21062,97,+++++...
```

This is just with just L2ARC (unencrypted):

```
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
zhenbox.home.do 96G   129  99 81017  15 51120  13   367  96 166962  15 134.7   2
Latency               264ms    3556ms    3512ms     121ms     382ms     998ms
Version  1.96       ------Sequential Create------ --------Random Create--------
zhenbox.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 28051  96 +++++ +++ 27384  98 24952  97 +++++ +++ 27824  98
Latency              9620us     604us     123us   28137us      25us     106us
1.96,1.96,zhenbox.home.domain,1,1337985771,96G,,129,99,81017,15,51120,13,367,96,166962,15,134.7,2,16
,,,,,28051,96,+++++,+++,27384,98,24952,97,+++++...
```

In fact, one thing I noticed is that results seem to be a bit better without (!) L2ARC on SSD (about 10-20MBps better reads) on my mirror vdevs. Why is this? And does this mean that mirror vdevs can only benefit from super fast L2ARC or something? This was unexpected since I figured L2ARC should make this a bit faster, even with a bit higher latency from encryption. Or maybe this just isn't good test.

On ZIL, I was monitoring activity with *zpool iostat -v 1* and it always looked like ZIL gets filled up to 200MB or so, to, both *geli* encrypted or not. Test showed that I get maybe 10MBps extra on sequential writes (if ZIL is unencrypted). 

All in all, what would be the best way to test this again and see if this setup even benefits from this SSD? Does it make sense to put both ZIL and L2CARC on this SSD as they are likely to comete for resources? Are there tunable that should be set (*noprefetch* is already set to 0)? I guess limiting factor to this is *geli* but I'm unable to test it unencrypted at this time.

Lot's of questions, so whatever you could answer, thanks.


----------



## t1066 (May 28, 2012)

You are correct that L2ARC must be pointed to by ARC. This is why we need to warm up the L2ARC again after each reboot. I am not sure how difficult to reconstruct files or part of files from a bunch of mostly 128KB sized blocks. But still if there are sensitive information in plain text files, then these may be glimpse from a raw L2ARC. Also, speed wise, a geli encrypted cache drive is limited to a reading speed of 200MB/s on a E3-1230.

I think that bonnie++ is not a reliable way to test the benefit of a L2ARC. First, it may take a long time for the L2ARC to warm up properly. Then, it create a workload that only last a few hours. Finally, if you run L2ARC and SLOG on the same SSD, it may adversely affect the performance of the L2ARC. As an example when I start a VM the first time, it takes around 1 minute for the whole thing to boot, while starting the same VM the next day (when most ARC related to the VM are evicted) takes less than 20 seconds. 

I will use the following example to illustrate how I would use the L2ARC. I want to set up a server to distribute different branches of FreeBSD. My goal is that most of the files will be served from the L2ARC and the spindles will be idle most of the time. This will lower the overall power consumption and reduce latency of the pool. Now suppose that there is 12GB RAM for ARC and assuming the size of each branch of FreeBSD (including iso, packages etc.) is exactly 12GB. There is only one 60GB SSD to be added to the pool as a cache drive. Now I create two filesystems, freebsd-new and freebsd-old. I put the 8.x branches and the 9.0 branch onto freebsd-new and set secondarycache=all, while older branches are put onto freebsd-old with secondarycache=none. Using zfs-stats and zstat to monitor the usage and efficiencies of ARC and L2ARC, I could check whether this strategy make sense or not, what other tuning that may be necessary. And it may take a few days for me to get my answers.

On ZIL, it is always on the pool. It may be on the main pool, or on a SLOG (separate log device). Its use is to replay last sync operation when the pool crashes (to guarantee POSIX compliant). You may notice it will suddenly grow from a few MB to several hundred MB and then back to a few MB.

For future testing, I would suggest you first determine your goal, then devise a way to achieve it. And most importantly, test L2ARC and SLOG one at a time.


----------



## bbzz (May 28, 2012)

Thank you for taking time to answer. I'll have to look more into it. Is there a specification for how much limiting *geli* is on that CPU or did you do test yourself?

I guess unencrypted L2ARC can be a big benefit to completely encrypted system if data server is not confidential, as in example you provided.


----------



## t1066 (May 28, 2012)

Just out of curiosity, I tested running ZFS on a PLEXTOR PX-128M2P. Run bonnie++ without and with encryption gave


```
Version      1.96   ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aaa.local       32G   203  99 272466  38 169484  29   538  99 458453  32  2301  72
Latency               139ms   21429us     643ms   18865us     244ms   32009us
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
aaa.local        16 32035  95 +++++ +++ +++++ +++ 32243  98 +++++ +++ +++++ +++
Latency             11082us      87us      99us   21944us      24us      56us
```


```
Version      1.96   ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aaa.local       32G   212  99 160276  21 86818  13   528  99 192479  12  1869  17
Latency               185ms     827ms    2452ms   23599us     355ms     216ms
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
aaa.local        16 32383  92 +++++ +++ +++++ +++ 31316  97 +++++ +++ 32624  98
Latency             12228us     100us     793us   23288us      32us     815us
```


----------

