# ZFS: How To Calulate for Tuning?



## dave (Dec 8, 2009)

I have looked at the tuning guide, and I am still pretty confused.  I guess you have to know a lot about FreeBSD internals to be able to know how much kmem, arc, etc to configure?  I am pretty good with FreeBSD, but this is totally confusing to me.

For instance, consider the following:  "The following script will summarize kernel memory utilization and assist in tuning arc_max and VDEV cache size."  I run the script, but what is it supposed to be telling me about how to configure /boot/loader.conf?

I have 2GB of RAM

uname -a

```
[dave@tank ~]$ uname -a
FreeBSD tank.xxx.xxx 8.0-RELEASE FreeBSD 8.0-RELEASE #1: Fri Nov 27 13:26:50 PST 2009     root@tank.xxx.xxx:/usr/obj/usr/src/sys/TANKZFS  i386
```

memory monitor script ouput

```
[dave@tank ~]$ zmem.sh 
TEXT=13189088, 12.5781 MB
DATA=47984640, 45.7617 MB
TOTAL=61173728, 58.3398 MB
```

cpu:

```
AMD Athlon(tm) XP 2800+ (2083.11-MHz 686-class CPU)
```

kernel conf:

```
#
# TANKZFS -- Kernel conf file for Tank with ZFS Tuning by DT on 20091113
#
include		GENERIC
ident		TANKZFS

options		KVA_PAGES=512
```

/boot/loader.conf

```
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="320M"
vfs.zfs.vdev.cache.size="5M"
vfs.zfs.prefetch_disable=1
```

I have 3 750GB drives configured in a ZRAID.

I have tried many different combinations in the /boot/loader.conf file.

Whenever my system does anything on more than few files on my zfs, the system just reboots.  For instance, the daily cron check setiud causes the machine to reboot.  So, this is not stable at all.

Am I missing something?  Is this machine not good enough for ZFS?  Is there a way to calculate how to configure /boot/loader.conf?


----------



## graudeejs (Dec 8, 2009)

I'd say remove *vfs.zfs.vdev.cache.size="5M"* and *vfs.zfs.prefetch_disable=1*
and set *vfs.zfs.arc_max="512M"*

prefetch is disabled by default on i386

here's my settings for 2.5G ram

```
vm.kmem_size="1100M"
vm.kmem_size_max="1100M"
vfs.zfs.arc_max="600M"
```


Note that you may need to create custom kernel.
In my custom kernel I have

```
options     KVA_PAGES=384
```
you may need to adjust this number... with many reboots and panics...
I would like to set it higher, but I can't..... because reboots...

If I remember correctly you must be able to divide KVA_PAGES by 4

Hope this will help you


----------



## wonslung (Dec 8, 2009)

I'm FAIRLY sure freebsd 8.0 can run fine without tuning....i know for SURE it can on amd64....not 100% sure about 32bit though.


what i would do is this:

install freebsd and set up raidz like you want...then install something like iozone and run a few tests on it.

i would imagine your system should be fine for ZFS, though it's kinda on the low end....i know people who have run it with a p4 and only 1 GB ram, though i haven't PERSONALLY done this.


----------



## graudeejs (Dec 8, 2009)

AMD64 does autotuning.... i386 doesn't


----------



## wonslung (Dec 8, 2009)

yah, i see that now after reading up on it (this post got me interested in it)

I remember checking up on it when i set my system but but i must not have read it as well as i should have....I wasn't 100% sure...thanks for the info.


----------



## dave (Dec 9, 2009)

What is the script in the tuning guide supposed to tell me about how to tune for ZFS?  Here is my output:


```
TEXT=13189088, 12.5781 MB
DATA=109240320, 104.18 MB
TOTAL=122429408, 116.758 MB
```

I have seen it get to maybe 250MB TOTAL before a panic.  Does this somehow help me know what numbers to put into the /boot/loader.conf file?  What is the relationship?


----------



## graudeejs (Dec 9, 2009)

It's all about testing and restarting.... until it works good


----------



## phoenix (Dec 9, 2009)

My rule of thumb has been to set *vm.kmem_max_size* to half of RAM, and set *vfs.zfs.arc_max* to half of vm.kmem_max_size.

With the caveat that 32-bit FreeBSD up to 7.1 has a kmem_max_size limit of 1596 MB, and 32-bit FreeBSD 7.2 has a kmem_max_size_limit of about 2.5 GB.

That's a good starting place.  And then you just play with (reduce) the arc_max setting whenever the system locks up.  

If possible, even if you have less than 4 GB of RAM, run 64-bit FreeBSD.  You'll get fewer memory fragmentation and exhaustion issues.


----------



## graudeejs (Dec 9, 2009)

Hmm, Phoenix... do you recommend minimizing arc, when possible? [I'm not clear, what you mean]


----------



## phoenix (Dec 9, 2009)

Yeah, each time it crashes, you reduce the ARC max size a bit.  Eventually, you'll find a setting that works "perfectly".

On my home server, which is 32-bit, I have arc_max set to 128M, for example.


----------



## graudeejs (Dec 10, 2009)

hmmm, my curent acr_max is 600M.... I have 2.5G ram on i686 arch
I managed to compile all OOO packages with this {I think it's good stress test. lol}

So, would you then suggest me to increase it untill I find point, when it starts to crash, and then go 1 step back?

This is interesting....


P.S.

```
vm.kmem_size="1100M"
vm.kmem_size_max="1100M"
vfs.zfs.arc_max="600M"
```


----------



## phoenix (Dec 10, 2009)

That's pretty much the process on 32-bit systems.  Change a setting, run until it crashes.  Change a setting, run until it crashes.  Repeat until it takes several weeks/months to crash.    Each workload will have its own "sweetspot".


----------



## dave (Oct 3, 2010)

I should note that while ZFS tuning was a bit troublesome, I later found out that there was another problem which was exacerbating the situation.  I had a corrupt folder on the ZFS pool that was crashing the system every time clamav tried to scan it.  I thought that the machine was rebooting due to performance/memory issues with ZFS, and in some cases it was.  But I was closer than I thought to a stable config.


----------

