# Finding what is writing to my zpool (dtrace perhaps?)



## jkcarrol (Jun 24, 2011)

I have a RAIDZ1 zpool consisting of three 1TB WD Black drives, and I'm seeing some periodic writes to the pool that I'd like to track down. It's usually a few hundred KB/s or up to 1-2 MB/s according to *zpool iostat* and I've had no luck watching *top -m io -s .5* to look for an I/O intensive process. As far as I know, the zpool should be pretty much idle. So I'm just curious what these periodic writes are.

The zpool in question consists of the following mounts/ds:


```
data                       514G  1.28T  58.6K  /data
data/home                 7.04G  1.28T  1.21G  /usr/home
data/home_video           31.5G  1.28T  31.5G  /data/home_video
data/jails                2.65G  1.28T  2.65G  /usr/jails
data/music                48.2G  1.28T  48.1G  /data/music
data/pictures             86.2G  1.28T  85.6G  /data/pictures
data/usr_local            5.79G  1.28T  4.82G  /usr/local
data/var                   887M  1.28T   369M  /var
```

The only processes/file systems that aren't entirely idle are the following:


data/home - my user's home directory. The only active processes with anything open are irssi inside a tmux session and a couple of eggdrops. None of the files open for write on here are writing more than a few bytes every few seconds. Certainly not hundreds of KB/s or MB/s
data/pictures - fed via web server for a gallery. Should be *only reads* (uploads are disabled except for admins).
data/usr_local - nothing writing here other than the occasional port update. certainly nothing writing here every few seconds
data/var - the usual logs/etc being written here, but nothing constituting MB/s of writes. I have smokeping running update two rrd files, but again we're talking bytes or 10's of KB, not MB/s.

I went ahead and built a kernel with proper dtrace support and I was hoping someone might know of a way to use it to see what encompasses these writes. This is my first time fiddling with dtrace, so I'm not yet sure if it's even possible with dtrace in 8.2-RELEASE.

Another piece of data is that usually the number of writes is about 100 or so, so it seems to be something that's fairly consistent in whatever it is writing.

Any help (dtrace or otherwise) in tracking down these writes would be greatly appreciated.

Thanks in advance! Here is a snapshot of *zpool iostat* to give you an idea of what I'm seeing:


```
# zpool iostat data 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
data         772G  1.96T      6     21  67.8K   112K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    106      0   463K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    279      0  1.81M
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    130      0   507K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    103      0   254K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    105      0   335K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    108      0   475K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    105      0   338K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0   128K
data         772G  1.96T      0    133      0   628K
```


----------



## usdmatt (Jun 24, 2011)

If you're brave enough to move to 8-STABLE and upgrade to ZFS v28 you could take a couple of snapshots and do a *zfs diff*.

You can also take a snapshot and run the following to find files modified. (Posted by Alt in the following thread about finding differences in ZFS filesystems - http://forums.freebsd.org/showthread.php?t=10883)

[cmd=]find /mnt/store1/ -newer /mnt/store1/.zfs/snapshot/yesterday -type f[/cmd]

I don't know anything about dtrace but no-one else has answered so I thought I'd throw a few ideas out there...


----------



## AndyUKG (Jun 24, 2011)

Hi,

  I think rwsnoop should do what you want, should be included in the DTraceToolkit port:

http://www.freshports.org/sysutils/DTraceToolkit/

Not tried it though, feel free to report back to enlighten me and anyone else reading 

cheers Andy.


----------



## jkcarrol (Jun 24, 2011)

usdmatt said:
			
		

> If you're brave enough to move to 8-STABLE and upgrade to ZFS v28 you could take a couple of snapshots and do a *zfs diff*.



Excellent! I'm using mm's zfs v28 patch on 8.2-RELEASE so I will definitely try this. I've already got some daily snapshots, but obviously would need some taken closer to each other (and when my other machines' backups aren't running). Thanks for the idea though!


----------



## jkcarrol (Jun 24, 2011)

AndyUKG said:
			
		

> Hi,
> 
> I think rwsnoop should do what you want, should be included in the DTraceToolkit port:
> 
> ...



Thanks for the tip! I installed the port, but the sh-bang for the rwsnoop script is using ksh (which I lack) and it doesn't want to work properly with /bin/sh nor with /usr/local/bin/zsh. I'll fiddle with it later today and try using the ksh93 port, but thanks for the info. This definitely looks promising!


----------



## AndyUKG (Jun 24, 2011)

Yeah you will need ksh, just having a little look at the handbook where it states:



> support for the Korn shell should be added. This is needed as the DTrace toolkit has several utilities written in ksh. Install the shells/ksh93. It is also possible to run these tools under shells/pdksh or shells/mksh.



In case you didn't know dtrace support isn't in the default GENERIC kernel, so if you didn't already you will have to recompile with the relevant dtrace options enabled. Shame it's not in by default IMO, maybe there are reason for that...?

Andy.


----------



## jkcarrol (Jun 24, 2011)

AndyUKG said:
			
		

> Yeah you will need ksh, just having a little look at the handbook where it states:



Whoops, didn't actually read the section in the handbook on it (honestly, didn't know there was a chapter on it, sweet!), sorry. Why not make ksh93 a runtime dependency for the port? I guess I'll contact the maintainer with a patch. Seems like it'd make sense to install it or at least make it toggle-able with a knob. I'll play with this tonight, thanks again for everyone's input! 




> In case you didn't know dtrace support isn't in the default GENERIC kernel, so if you didn't already you will have to recompile with the relevant dtrace options enabled. Shame it's not in by default IMO, maybe there are reason for that...?
> 
> Andy.



Right, I have already built a proper kernel and world per the DTrace wiki, and the examples I've tried seem to work fine. Thanks though!


----------

