# Effects of scrub on VFS read latency



## t1066 (Feb 9, 2013)

VFS(), Virtual file system is where applications interact with the underlying storage system. Hence, the responsiveness of an application depends more on latency of this layer than the actual latency of the underlying disks.

Just for curiosity, I have measured the VFS read latency before, during and after scrub using the following dtrace script.


```
vfs::vop_read:entry 
{ 
    self->st[stackdepth] = timestamp; 
} 

vfs::vop_read:return 
/self->st[stackdepth]/ 
{ 
    @ = quantize(timestamp - self->st[stackdepth]);
    self->st[stackdepth] = 0; 
}

tick-1200s
{
    exit(0);
}
```

The system tested was an E3-1230 with 16GB RAM, 6 disks in a raidz2 pool, 1 256GB L2ARC and 1 5GB SLOG.

Before scrub, the latencies in ns


```
value  ------------- Distribution ------------- count
             512 |                                         0
            1024 |                                         84
            2048 |                                         950
            4096 |                                         893
            8192 |@@@@                                     19073
           16384 |@@                                       10824
           32768 |@@@@@@@@@@@@@@@@@@@@@                    93553
           65536 |@@@@@@@@                                 36897
          131072 |@                                        3445
          262144 |                                         1480
          524288 |@@                                       10453
         1048576 |@                                        2747
         2097152 |                                         695
         4194304 |                                         288
         8388608 |                                         120
        16777216 |                                         55
        33554432 |                                         4
        67108864 |                                         0
```

During scrub,


```
value  ------------- Distribution ------------- count
             256 |                                         0
             512 |                                         10
            1024 |                                         482
            2048 |@                                        3718
            4096 |@@@@                                     30482
            8192 |@@@@@@@@@@@@@@@@@@                       128855
           16384 |@@@@@@@@@                                63613
           32768 |@@@@@@                                   41409
           65536 |@                                        6978
          131072 |                                         2065
          262144 |@                                        5404
          524288 |@                                        9339
         1048576 |                                         1107
         2097152 |                                         472
         4194304 |                                         35
         8388608 |                                         3
        16777216 |                                         3
        33554432 |                                         10
        67108864 |                                         14
[U]       134217728 |                                         6
       268435456 |                                         6
       536870912 |                                         2
      1073741824 |                                         0[/U]
```

After scrub,


```
value  ------------- Distribution ------------- count
             512 |                                         0
            1024 |                                         121
            2048 |                                         877
            4096 |@@                                       11277
            8192 |@@@@@@                                   41507
           16384 |@@@@                                     26238
           32768 |@@@@@@@@@@@@@@@@@@@@                     136957
           65536 |@@@@@@                                   39477
          131072 |@                                        3511
          262144 |                                         2129
          524288 |@@                                       12421
         1048576 |                                         1805
         2097152 |                                         635
         4194304 |                                         125
         8388608 |                                         40
        16777216 |                                         27
        33554432 |                                         1
        67108864 |                                         3
       134217728 |                                         0
```

Before and after scrub, there were just a number of cases in which latency went higher than 10 ms. Still, it may make an application feels slow. However, during scrub, the maximum latency could reach as high as 1 s, which would surely affect responsiveness of some applications.


----------

