# Lag in local ZFS reads leading to slow application start



## yggdrasil (Jun 7, 2015)

Haven't found a better title, suggestions welcome.

My problem is, that after I got my new computer (quite some time ago already) and installed 10.1 on it, I got around setting up my `zsh` terminal and `vim` with fancy stuff. For example using powerline with git integration in `zsh`, in tmux, and in `vim`, several coding plugins in `vim` etc. What happened was a noticeable slowdown in several places, like the startup of `vim`, or the generation of the prompt in my terminal. Now, some of this I thought was due to powerline and its python backend. This seemed to be corroborated by internet searches for powerline problems, and by top sometimes showing about 15 instances of python running seemingly generating some prompt. But even when I switched to airline in `vim` which is implemented completely internally, `vim` would still start up very slow. So of course I thought, some plugin in my `vim`rc must be at fault, but even after disabling all of them, reenabling them one by one I got very erratic behaviour in startup times. It even goes so far as to sometimes start up almost instantaneously with the full plugin set loaded, but very slow the moment before when I had almost all turned off. 
Another thing I noticed was sometime very slow (meaning 10-15 seconds) autocompletion in my `zsh` prompt, esp. when it comes to completing ZFS snapshots, but in some instances even completing regular files and directories. This is corroborated when I do a `ls` in a .zfs/snapshot directory. The first time took forever to list about 100 snapshots, the second `ls` was instantaneous.
Sometimes I see quite a lot calls to `zfs list -H ...` in top, which probably comes from `zfstools` autosnapshotting, but they run forever and sometimes max out my cpu cores, ramping up my load average to over 10.
I think my record load avg was about 60 without really doing anything... But sometimes the system slows down without load avg or cpu usage going up, or just one of the two. RAM is practically never maxed out.

All of this happens whether my system is installed on an old HDD, or a new SSD.
So I think the core of the problem is disk access via ZFS, however I don't know what exactly to do about it and would appreciate any help.

Just wanted to paste my `dmesg` output here, but it outputs nothing?! WTF?!

8GB RAM ECC
AMD FX-4100 on an Asus M5A97 R2.0 970 RG SA

/boot/loader.conf

```
geli_ada1p4_keyfile0_load="YES"
geli_ada1p4_keyfile0_type="ada1p4:geli_keyfile0"
geli_ada1p4_keyfile0_name="/boot/encryption.key"
aesni_load="YES"
geom_eli_load="YES"
vfs.root.mountfrom="zfs:midgard/ROOT/default"
zfs_load="YES"
vfs.zfs.arc_max=1073741824
nvidia_load="YES"
kern.geom.label.gptid.enable="0"
zpool_cache_load="YES"
zpool_cache_type="/boot/zfs/zpool.cache"
zpool_cache_name="/boot/zfs/zpool.cache"
vboxdrv_load="YES"

fdescfs_load="YES"
fuse_load="YES"

kern.vty=vt
hw.vga.textmode=1
kern.vt.fb.default_mode="1280x800"
kern.ipc.shmall=32768
kern.ipc.shmmni=1024
kern.ipc.shmseg=1024
```

sysctl.conf

```
compat.linux.osrelease=2.6.18
hw.syscons.bell=0

vfs.usermount=1

net.inet.ip.random_id=1
```


----------

