# ZFS panics/tuning



## Voltar (Jun 30, 2009)

So after all the information from my last thread about storage, Iâ€™ve decided to use ZFS on my main file server, and transfer everything over slowly from their existing XFS filesystems to a RAIDZ. Iâ€™ve had ZFS and RAIDZ going for awhile now with no issues, so I started to do some benchmarking and stress testing to make sure that I donâ€™t end up with a kernel panic when starting to move a lot of files via smb/nfs/ftp/rsync.

Originally when I had 7.1 on my other box (said box from the other thread, x2 3800+, 4GB RAM and 7.1 amd64), I had it tuned and it worked well, no kernel panics or anything. I'm using this box for testing because I don't want to end up with something unusable after I start slowly putting the 12 drives in my FS in the new box. For 7.1 I had this in /boot/loader.conf


```
vm.kmem_size="1536M"
vm.kmem_size_max="1536M"
vfs.zfs.arc_max="100M"
```

And I had kern.maxvnodes=500000 in sysctl.conf.


Now when I upgraded to 7.2, I just blatantly ignored the addendum on the wiki about 7.2 having autotuning because I figured if it isn't broke, don't fix it. Here's when the trouble came in. I tried running iozone to get a good idea of the performance I'd see, and if I'd get a panic with a lot of disk activity. 

I lost the original iozone commands I was using when the box kp'd but it was similar to this:


```
iozone -R -l 5 -u 5 -r 4M -s 256M
```

That will panic the box with the aforementioned settings or letting the box tune itself. 

Something more conservative like: 


```
iozone -R -l 5 -u 5 -r 4M -s 100M
```

Always completes successfully, as with a simple (took it off some site):


```
iozone -ec -t 1 -r 4M -s 100M -+n -i 0 -i 1
```


The panic is always something like this, with X being the number of bytes I allocated in loader.conf:


```
panic: kmem_malloc(131072): kmem_map too small: X total allocated
```


I'm basically wondering if I should bump up the kmem_sizes, or if I'm just missing another 'magical' setting somewhere. Also, I don't use GENERIC. 



If it helps, in the end if I can overcome the panics, the file server I'd be installing on is an AMD Opteron 170, 4GB RAM, amd64 7.2, and something on the order of 12x500 GB drives and 4x250.


----------



## Voltar (Jul 1, 2009)

I've tweaked the settings in loader.conf a bit more now and I currently have this along with the previous sysctl value I mentioned: 


```
vm.kmem_size="1536M"
vm.kmem_size_max="2048M"
vfs.zfs.arc_max="80M"
```

And I've ran all my previous tests successfully. 

I also used the kernel memory utilization script in the ZFS Tuning Guide on the wiki, and the total never went over 500MB with the above settings, and my vfs.numvnodes never went over 2000. I've still left vfs.zfs.vdev.cache.size at it's default which I believe is 10 MB. 

Maybe I'm just worrying about this too much?


----------



## phoenix (Jul 3, 2009)

The max value for kmem in 64-bit FreeBSD 7.2+ is now 5 GB (max in 7.1 was ~2 GB although most systems failed with kmem_size_max set above ~1596 MB).

Two things I've found in my "re-tuning" of our 64-bit ZFS boxes:

Using 1/2 RAM for kmem_size_max, and 1/2 of that for arc_max seems to be the most stable setup (for us),
Setting vfs.zfs.prefetch_disable=1 solves a lot of kmem exhaustion issues
If we enable prefetching, then we have to *really* limit the ARC (<512MB), even on systems with 8 GB of RAM.  The way the prefetch works is along the lines of "ask for 1 block, prefetch 2 blocks; if the next requested block is in the ARC already, then prefetch 4 blocks; if the next requested block is in the ARC already, then prefetch 8 blocks" and so on.  Which can quickly fill the ARC ... and easily go beyond the value of arc_max (which is just a guide for when to start the arc-cleaning thread).

Are you running with prefetch enabled or disabled?  If enabled, try disabling it and see how things go.


----------



## tanked (Jul 5, 2009)

Perhaps you may want to try upgrading to 7-STABLE which has ZFS v.13

I run 7-STABLE (32bit i386) on an Athlon 2 GHz with 1GB of memory using the kernel tunables in the FreeBSD handbook and I have had no problems so far. My setup is two drives in a mirrored configuration, its not heavily used but I been able to regularly copy 42GB of data (over 5000 files) to and from the ZFS file-system with no problems whatsoever.


----------



## tanked (Jul 5, 2009)

phoenix said:
			
		

> The max value for kmem in 64-bit FreeBSD 7.2+ is now 5 GB (max in 7.1 was ~2 GB although most systems failed with kmem_size_max set above ~1596 MB).
> 
> Two things I've found in my "re-tuning" of our 64-bit ZFS boxes:
> 
> ...



Do you still find that tunables are required for 64-bit FreeBSD 7.2?


----------



## Voltar (Jul 6, 2009)

phoenix said:
			
		

> Two things I've found in my "re-tuning" of our 64-bit ZFS boxes:
> 
> Using 1/2 RAM for kmem_size_max, and 1/2 of that for arc_max seems to be the most stable setup (for us),
> Setting vfs.zfs.prefetch_disable=1 solves a lot of kmem exhaustion issues



I tried the using the kmem_size_max and arc_max like you said and so far I got good results. I was previously running with prefetch enabled but went ahead and disabled it when I rebooted for the other settings to take effect. With the values as they are and prefetch disabled I can hammer the file system pretty hard, to the point I got tired of _trying_ to make it panic or lockup on me. 

I might try turning prefetch back on and seeing if I still have a stable system.




> If we enable prefetching, then we have to *really* limit the ARC (<512MB), even on systems with 8 GB of RAM.  The way the prefetch works is along the lines of "ask for 1 block, prefetch 2 blocks; if the next requested block is in the ARC already, then prefetch 4 blocks; if the next requested block is in the ARC already, then prefetch 8 blocks" and so on.  Which can quickly fill the ARC ... and easily go beyond the value of arc_max (which is just a guide for when to start the arc-cleaning thread).



Good info, very interesting. 




			
				tanked said:
			
		

> Perhaps you may want to try upgrading to 7-STABLE which has ZFS v.13



I'm actually thinking about that just to see if there are any improvements on my system.


----------



## phoenix (Jul 6, 2009)

tanked said:
			
		

> Do you still find that tunables are required for 64-bit FreeBSD 7.2?



Yes ... and no.  

I was running without any tuning in /boot/loader.conf for a couple of weeks.  Then I enabled prefetch, and things went downhill.  Tried tuning things a bunch of different ways, but couldn't get things stable again.  So I've disabled prefetch again, and have left the tuning enabled (2 GB for kmem, 1 GB for arc, 8 GB RAM system).

If there are no more lockups over the next week or two, then I'll try disabling the manual tuning again.

Seems like it's just prefetch that is causing the issues ... but will wait to be sure.


----------



## kbw (Jul 7, 2009)

How do you check kernel memory usage, for example the vfs.numvnodes value?

Thanks.


----------



## phoenix (Jul 7, 2009)

Personally, I don't.    Should we be?


----------



## kbw (Jul 7, 2009)

Above, Voltar said 





> and the total never went over 500MB with the above settings, and my vfs.numvnodes never went over 2000



How do you measure these values?


----------



## phoenix (Jul 7, 2009)

Using sysctl:
`$ sysctl vfs.numvnodes vfs.zfs`
`$ sysctl kern | grep kmem`


----------



## Voltar (Jul 7, 2009)

kbw said:
			
		

> Above, Voltar said
> 
> How do you measure these values?



Here it has this script under i386, Made it a bit easier to keep an eye on the memory usage. 



```
#!/bin/sh -

TEXT=`kldstat | awk 'BEGIN {print "16i 0";} NR>1 {print toupper($4) "+"} END {print "p"}' | dc`
DATA=`vmstat -m | sed -Ee '1s/.*/0/;s/.* ([0-9]+)K.*/\1+/;$s/$/1024*p/' | dc`
TOTAL=$((DATA + TEXT))

echo TEXT=$TEXT, `echo $TEXT | awk '{print $1/1048576 " MB"}'`
echo DATA=$DATA, `echo $DATA | awk '{print $1/1048576 " MB"}'`
echo TOTAL=$TOTAL, `echo $TOTAL | awk '{print $1/1048576 " MB"}'`
```


----------



## sprewell (Oct 20, 2009)

I was really getting annoyed with the stuttering performance of zfs v6 on my FreeBSD 7.2 desktop.  For example, a HD movie would stutter every couple seconds when accessed from zfs but play smoothly from a UFS2 filesystem on the same hard drive.  With a tip from one of the zfs porters and phoenix's post here, I set prefetch_disable in /boot/loader.conf and now it appears to be much better.


----------



## wonslung (Oct 21, 2009)

sprewell said:
			
		

> I was really getting annoyed with the stuttering performance of zfs v6 on my FreeBSD 7.2 desktop.  For example, a HD movie would stutter every couple seconds when accessed from zfs but play smoothly from a UFS2 filesystem on the same hard drive.  With a tip from one of the zfs porters and phoenix's post here, I set prefetch_disable in /boot/loader.conf and now it appears to be much better.



weird...i'm fairly sure i'm using default zfs sysctls with 7-STABLE from back in august and i run multiple hd movie streams no problem.

I have 4 hd movies playing via samba right now and never notice a problem....
heres my sysctls

```
vfs.zfs.arc_meta_limit: 431800960
vfs.zfs.arc_meta_used: 240145904
vfs.zfs.mdcomp_disable: 0
vfs.zfs.arc_min: 215900480
vfs.zfs.arc_max: 1727203840
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 0
vfs.zfs.recover: 0
vfs.zfs.txg.synctime: 5
vfs.zfs.txg.timeout: 30
vfs.zfs.scrub_limit: 10
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 6
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 35
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.version.zpl: 3
vfs.zfs.version.vdev_boot: 1
vfs.zfs.version.spa: 13
vfs.zfs.version.dmu_backup_stream: 1
vfs.zfs.version.dmu_backup_header: 2
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
kstat.zfs.misc.arcstats.hits: 443068884
kstat.zfs.misc.arcstats.misses: 62142723
kstat.zfs.misc.arcstats.demand_data_hits: 210881210
kstat.zfs.misc.arcstats.demand_data_misses: 16799446
kstat.zfs.misc.arcstats.demand_metadata_hits: 166396123
kstat.zfs.misc.arcstats.demand_metadata_misses: 12539719
kstat.zfs.misc.arcstats.prefetch_data_hits: 8246888
kstat.zfs.misc.arcstats.prefetch_data_misses: 30315977
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 57544663
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 2487581
kstat.zfs.misc.arcstats.mru_hits: 50873040
kstat.zfs.misc.arcstats.mru_ghost_hits: 22223445
kstat.zfs.misc.arcstats.mfu_hits: 326988764
kstat.zfs.misc.arcstats.mfu_ghost_hits: 14817882
kstat.zfs.misc.arcstats.deleted: 31315830
kstat.zfs.misc.arcstats.recycle_miss: 40021588
kstat.zfs.misc.arcstats.mutex_miss: 208494
kstat.zfs.misc.arcstats.evict_skip: 3460550618
kstat.zfs.misc.arcstats.hash_elements: 11914
kstat.zfs.misc.arcstats.hash_elements_max: 135004
kstat.zfs.misc.arcstats.hash_collisions: 8205729
kstat.zfs.misc.arcstats.hash_chains: 435
kstat.zfs.misc.arcstats.hash_chain_max: 8
kstat.zfs.misc.arcstats.p: 514034429
kstat.zfs.misc.arcstats.c: 525024585
kstat.zfs.misc.arcstats.c_min: 215900480
kstat.zfs.misc.arcstats.c_max: 1727203840
kstat.zfs.misc.arcstats.size: 524925936
kstat.zfs.misc.arcstats.hdr_size: 2670976
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 70930932
kstat.zfs.misc.vdev_cache_stats.delegations: 1340508
kstat.zfs.misc.vdev_cache_stats.hits: 23191278
kstat.zfs.misc.vdev_cache_stats.misses: 10827933
```


----------



## sprewell (Nov 26, 2009)

It's probably related to system memory then.  I'm underspeced for zfs with only 1 GB of RAM in the system, I bet you have more.


----------



## wonslung (Nov 27, 2009)

yes, i have 8gb
memory is dirt cheap...and ZFS loves memory.


----------

