# ZFS and Memory



## Leander (May 1, 2012)

Hi,

I also tried my luck with ZFS on

FreeBSD - 9.0-RELEASE 
IntelÂ® Coreâ„¢ i5 CPU 760 @ 2.80GHz (4xCore)
64 bit
8GB RAM
4GB file based swap

Now, obviously one can read *lots of things*, tips and tweaks for FreeBSD System in order to use ZFS "_proper_". Now, some of those things you'll fall over the most when reading are basically the following four sysctl nodes in order to get a stable system without kernel panics:


vm.kmem_size _Size of kernel memory_
vm.kmem_size_max _Maximum size of kernel memory_
vfs.zfs.arc_min _Minimum ARC size_
vfs.zfs.arc_max _Maximum ARC size_
I left those variables on its default values for my first try:


vm.kmem_size _7921.0 MB_
vm.kmem_size_max _314572.8 MB_
vfs.zfs.arc_min _862.1 MB_
vfs.zfs.arc_max _6897.0 MB_
... and my system got kernel panic after a 72G of heavy copying. Note that I do not have L2ARC.

Please correct me if I'm wrong here but as far as I understood those variables is that vfs.zfs.arc_* sets the frame/boundaries for the Layer 1 cache of ZFS. vfs.zfs.arc_min is what system should always keep full while vfs.zfs.arc_max represents buffer for higher loads. Also this Layer 1 cache depends and acts within the given frame of vm.kmem_size.
Basically to be on the same page here

Layer 1 cache ==> slice of system RAM
Layer 2 cache ==> Additional Storage Medium (i.e. SSD)

Now some people advice you to give vm.kmem_size upto 16GB on a 8GB RAM system. This sounds absurd to me since system can not even make use of 16GB since there is only 8GB RAM installed?! Or would this mean you also need a 8GB swap in order to not run out of pages here?

Basically my intuition tells me to let vm.kmem_size = Installed RAM. In fact I figured FreeBSD automatically defaults vm.kmem_size to 94% of hw.realmem - which sounds healthy - while the default vm.kmem_size_max = 314572.8 MB just doesn't sound healthy to me?! Why such a big number? Where should all this be stored in such a "worst case scenario"? Wouldn't it sound more reasonalbe to let vm.kmem_size_max = 94% of hw.realmem PLUS the amount of currently setup swap file's/drive's size?

Is there any reasons against my theory?

Anyway, after some tests I figured to get kernel panics under control by decreasing vfs.zfs.arc_max to 4G *(50% of my RAM)* and at the same time increase vfs.zfs.arc_min to 1G *(12.5% of my RAM)*.

Now fine, I don't have kernel Panics anymore (right now) - but note that at the same time there is a samba DC, apache, and some other services running on the system as well - and frankly I'm afraid that the system may run into a kernel panic again as soon as those service get some higher load ... ;/ - which is totally unacceptable for a productive system!

This doesn't give me butterflies in my stomach - in fact it makes me feel like FreeBSD's memory management is not well enough off to deal with high load, since I expect it to let a lower prio app wait until the higher prio one finished its work?! But I'm probably just missing out on the right sysctl variables to restrict/prevent kernel panics at all. Is there any suggestions for this based on facts? Or how Can I make sure other services don't disturb ZFS and the other way around and at the same time make *full* use of given RAM resources?

So how do we milk this cow? 

Thanks


----------



## Leander (May 1, 2012)

> ## KERNEL MEMORY
> #
> # Kernel memory is what ZFS uses and thus tuning this parameter may be useful.
> # General recommendation is to set this variable at 1.5 times your RAM size,
> ...



found on: http://zfsguru.com/forum/zfsgurusupport/82

Why more than physical RAM? I don't get it? Isn't that provoking a panic?


----------



## kpa (May 1, 2012)

The kmem* parameters are not actual amounts of memory but size parameters for a virtual address space where pages of memory (4KB units in i386 and amd64) can be mapped from the physical memory anywhere within the address range. This virtual address space will mostly consist of unused large gaps between small areas of used memory, that's why the address space can be just about as large you want even if you have small amount of physical memory (actually the amount of physical memory does limit the maximum size of this virtual address space because there has to be bookkeeping about which pages are in use and which ones are not). 

The idea is that pages of memory can be moved around and merged to larger continuous units in that virtual address space when memory is deallocated to avoid memory fragmentation.

Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.


----------



## Leander (May 1, 2012)

```
Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.
```
Thanks, this makes me feel more on the sa*f*e side too, since I've noticed kernel autotunes them according to the detected RAM amount.

Anyway, I've been testing the entire day under higher system load without any kernel panic thus far:

```
vm.kmem_size:        7,735 GB
vm.kmem_size_max:  307,2 GB
vfs.zfs.arc_min:     1,0 GB
vfs.zfs.arc_max:     5,0 GB
```


```
vfs.numvnodes: 56385
kern.maxvnodes: 206627
```

Since I consider myself a ZFS beginner it would be lovely if someone could give me hints on where to adjust more to the safe but still performant side. I found the arc_summary.pl script which should summarize the most important information. The script can be found at FreeBSD Wiki - ZFS Tuning Guide

Here is my output after a fresh reboot and before any read/write action on the ZFS:

```
arc_summary.pl 

------------------------------------------------------------------------
ZFS Subsystem Report				Tue May  1 16:27:56 2012
------------------------------------------------------------------------

System Memory:

	0.97%	77.10	MiB Active,	0.57%	44.99	MiB Inact
	2.52%	199.87	MiB Wired,	0.00%	280.00	KiB Cache
	95.93%	7.42	GiB Free,	0.01%	544.00	KiB Gap

	Real Installed:				8.00	GiB
	Real Available:			99.72%	7.98	GiB
	Real Managed:			96.96%	7.74	GiB

	Logical Total:				8.00	GiB
	Logical Used:			6.70%	548.53	MiB
	Logical Free:			93.30%	7.46	GiB

Kernel Memory:					98.66	MiB
	Data:				79.50%	78.44	MiB
	Text:				20.50%	20.22	MiB

Kernel Memory Map:				7.74	GiB
	Size:				0.37%	29.06	MiB
	Free:				99.63%	7.71	GiB
								Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
	Storage pool Version:			28
	Filesystem Version:			5
	Memory Throttle Count:			0

ARC Misc:
	Deleted:				7
	Recycle Misses:				0
	Mutex Misses:				0
	Evict Skips:				0

ARC Size:				0.03%	1.39	MiB
	Target Size: (Adaptive)		100.00%	5.00	GiB
	Min Size (Hard Limit):		20.00%	1.00	GiB
	Max Size (High Water):		5:1	5.00	GiB

ARC Size Breakdown:
	Recently Used Cache Size:	50.00%	2.50	GiB
	Frequently Used Cache Size:	50.00%	2.50	GiB

ARC Hash Breakdown:
	Elements Max:				166
	Elements Current:		100.00%	166
	Collisions:				0
	Chain Max:				0
	Chains:					0
								Page:  2
------------------------------------------------------------------------

ARC Efficiency:					5.65k
	Cache Hit Ratio:		96.92%	5.48k
	Cache Miss Ratio:		3.08%	174
	Actual Hit Ratio:		95.47%	5.40k

	Data Demand Efficiency:		100.00%	0

	CACHE HITS BY CACHE LIST:
	  Anonymously Used:		1.50%	82
	  Most Recently Used:		14.29%	783
	  Most Frequently Used:		84.22%	4.62k
	  Most Recently Used Ghost:	0.00%	0
	  Most Frequently Used Ghost:	0.00%	0

	CACHE HITS BY DATA TYPE:
	  Demand Data:			0.00%	0
	  Prefetch Data:		0.00%	0
	  Demand Metadata:		98.39%	5.39k
	  Prefetch Metadata:		1.61%	88

	CACHE MISSES BY DATA TYPE:
	  Demand Data:			0.00%	0
	  Prefetch Data:		0.00%	0
	  Demand Metadata:		51.72%	90
	  Prefetch Metadata:		48.28%	84
								Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:					6.25k
	Hit Ratio:			99.81%	6.24k
	Miss Ratio:			0.19%	12

	Colinear:				12
	  Hit Ratio:			0.00%	0
	  Miss Ratio:			100.00%	12

	Stride:					6.16k
	  Hit Ratio:			100.00%	6.16k
	  Miss Ratio:			0.00%	0

DMU Misc:
	Reclaim:				12
	  Successes:			0.00%	0
	  Failures:			100.00%	12

	Streams:				82
	  +Resets:			0.00%	0
	  -Resets:			100.00%	82
	  Bogus:				0
								Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
	kern.maxusers                           384
	vm.kmem_size                            8305745920
	vm.kmem_size_scale                      1
	vm.kmem_size_min                        0
	vm.kmem_size_max                        329853485875
	vfs.zfs.l2c_only_size                   0
	vfs.zfs.mfu_ghost_data_lsize            0
	vfs.zfs.mfu_ghost_metadata_lsize        0
	vfs.zfs.mfu_ghost_size                  0
	vfs.zfs.mfu_data_lsize                  0
	vfs.zfs.mfu_metadata_lsize              168960
	vfs.zfs.mfu_size                        175104
	vfs.zfs.mru_ghost_data_lsize            0
	vfs.zfs.mru_ghost_metadata_lsize        4096
	vfs.zfs.mru_ghost_size                  4096
	vfs.zfs.mru_data_lsize                  0
	vfs.zfs.mru_metadata_lsize              535040
	vfs.zfs.mru_size                        1063424
	vfs.zfs.anon_data_lsize                 0
	vfs.zfs.anon_metadata_lsize             0
	vfs.zfs.anon_size                       0
	vfs.zfs.l2arc_norw                      1
	vfs.zfs.l2arc_feed_again                1
	vfs.zfs.l2arc_noprefetch                1
	vfs.zfs.l2arc_feed_min_ms               200
	vfs.zfs.l2arc_feed_secs                 1
	vfs.zfs.l2arc_headroom                  2
	vfs.zfs.l2arc_write_boost               8388608
	vfs.zfs.l2arc_write_max                 8388608
	vfs.zfs.arc_meta_limit                  1342177280
	vfs.zfs.arc_meta_used                   1452568
	vfs.zfs.arc_min                         1073741824
	vfs.zfs.arc_max                         5368709120
	vfs.zfs.dedup.prefetch                  1
	vfs.zfs.mdcomp_disable                  0
	vfs.zfs.write_limit_override            0
	vfs.zfs.write_limit_inflated            25698410496
	vfs.zfs.write_limit_max                 1070767104
	vfs.zfs.write_limit_min                 33554432
	vfs.zfs.write_limit_shift               3
	vfs.zfs.no_write_throttle               0
	vfs.zfs.zfetch.array_rd_sz              1048576
	vfs.zfs.zfetch.block_cap                256
	vfs.zfs.zfetch.min_sec_reap             2
	vfs.zfs.zfetch.max_streams              8
	vfs.zfs.prefetch_disable                0
	vfs.zfs.mg_alloc_failures               8
	vfs.zfs.check_hostid                    1
	vfs.zfs.recover                         0
	vfs.zfs.txg.synctime_ms                 1000
	vfs.zfs.txg.timeout                     5
	vfs.zfs.scrub_limit                     10
	vfs.zfs.vdev.cache.bshift               16
	vfs.zfs.vdev.cache.size                 0
	vfs.zfs.vdev.cache.max                  16384
	vfs.zfs.vdev.write_gap_limit            4096
	vfs.zfs.vdev.read_gap_limit             32768
	vfs.zfs.vdev.aggregation_limit          131072
	vfs.zfs.vdev.ramp_rate                  2
	vfs.zfs.vdev.time_shift                 6
	vfs.zfs.vdev.min_pending                4
	vfs.zfs.vdev.max_pending                10
	vfs.zfs.vdev.bio_flush_disable          0
	vfs.zfs.cache_flush_disable             0
	vfs.zfs.zil_replay_disable              0
	vfs.zfs.zio.use_uma                     0
	vfs.zfs.version.zpl                     5
	vfs.zfs.version.spa                     28
	vfs.zfs.version.acl                     1
	vfs.zfs.debug                           0
	vfs.zfs.super_owner                     0
								Page:  7
------------------------------------------------------------------------
```


----------



## Leander (May 1, 2012)

... Output of arc_summary.pl right after I started cp to copy the content of a 2TB HDD over to my new ZFS:


```
------------------------------------------------------------------------
ZFS Subsystem Report                            Tue May  1 18:08:44 2012
------------------------------------------------------------------------

System Memory:

        1.67%   132.02  MiB Active,     38.03%  2.94    GiB Inact
        36.19%  2.80    GiB Wired,      0.00%   296.00  KiB Cache
        24.10%  1.86    GiB Free,       0.01%   644.00  KiB Gap

        Real Installed:                         8.00    GiB
        Real Available:                 99.72%  7.98    GiB
        Real Managed:                   96.96%  7.74    GiB

        Logical Total:                          8.00    GiB
        Logical Used:                   39.92%  3.19    GiB
        Logical Free:                   60.08%  4.81    GiB

Kernel Memory:                                  1.86    GiB
        Data:                           98.94%  1.84    GiB
        Text:                           1.06%   20.22   MiB

Kernel Memory Map:                              7.73    GiB
        Size:                           23.01%  1.78    GiB
        Free:                           76.99%  5.95    GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Storage pool Version:                   28
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                7
        Recycle Misses:                         0
        Mutex Misses:                           0
        Evict Skips:                            0

ARC Size:                               35.18%  1.76    GiB
        Target Size: (Adaptive)         100.00% 5.00    GiB
        Min Size (Hard Limit):          20.00%  1.00    GiB
        Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       50.00%  2.50    GiB
        Frequently Used Cache Size:     50.00%  2.50    GiB

ARC Hash Breakdown:
        Elements Max:                           18.11k
        Elements Current:               100.00% 18.11k
        Collisions:                             3.89k
        Chain Max:                              3
        Chains:                                 1.14k
                                                                Page:  2
------------------------------------------------------------------------

ARC Efficiency:                                 998.22k
        Cache Hit Ratio:                86.90%  867.43k
        Cache Miss Ratio:               13.10%  130.79k
        Actual Hit Ratio:               62.03%  619.25k

        Data Demand Efficiency:         99.97%  6.71k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             28.61%  248.19k
          Most Recently Used:           54.80%  475.31k
          Most Frequently Used:         16.59%  143.93k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  0.77%   6.70k
          Prefetch Data:                0.00%   0
          Demand Metadata:              70.61%  612.53k
          Prefetch Metadata:            28.61%  248.20k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.00%   2
          Prefetch Data:                0.00%   0
          Demand Metadata:              95.87%  125.39k
          Prefetch Metadata:            4.12%   5.39k
                                                                Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 3.62m
        Hit Ratio:                      81.64%  2.95m
        Miss Ratio:                     18.36%  664.16k

        Colinear:                               664.16k
          Hit Ratio:                    0.03%   195
          Miss Ratio:                   99.97%  663.97k

        Stride:                                 2.93m
          Hit Ratio:                    100.00% 2.93m
          Miss Ratio:                   0.00%   0

DMU Misc:
        Reclaim:                                663.97k
          Successes:                    0.15%   1.00k
          Failures:                     99.85%  662.97k

        Streams:                                22.89k
          +Resets:                      0.00%   0
          -Resets:                      100.00% 22.89k
          Bogus:                                0
                                                                Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            8305745920
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            0
        vfs.zfs.mfu_ghost_metadata_lsize        0
        vfs.zfs.mfu_ghost_size                  0
        vfs.zfs.mfu_data_lsize                  59343360
        vfs.zfs.mfu_metadata_lsize              173056
        vfs.zfs.mfu_size                        59522560
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        2449408
        vfs.zfs.mru_ghost_size                  2449408
        vfs.zfs.mru_data_lsize                  1709782528
        vfs.zfs.mru_metadata_lsize              8559616
        vfs.zfs.mru_size                        1722849792
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       95635456
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  1342177280
        vfs.zfs.arc_meta_used                   27272600
        vfs.zfs.arc_min                         1073741824
        vfs.zfs.arc_max                         5368709120
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            25698410496
        vfs.zfs.write_limit_max                 1070767104
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
                                                                Page:  7
------------------------------------------------------------------------
```


----------



## Leander (May 1, 2012)

And eventually my output of arc_summary.pl couple of hours later - still the same action - writing random files without - interruption of the copy process:

```
------------------------------------------------------------------------
ZFS Subsystem Report                            Tue May  1 21:05:33 2012
------------------------------------------------------------------------

System Memory:

        3.57%   282.61  MiB Active,     64.30%  4.97    GiB Inact
        27.30%  2.11    GiB Wired,      4.67%   370.14  MiB Cache
        0.15%   11.93   MiB Free,       0.01%   628.00  KiB Gap

        Real Installed:                         8.00    GiB
        Real Available:                 99.72%  7.98    GiB
        Real Managed:                   96.96%  7.74    GiB

        Logical Total:                          8.00    GiB
        Logical Used:                   33.17%  2.65    GiB
        Logical Free:                   66.83%  5.35    GiB

Kernel Memory:                                  1.08    GiB
        Data:                           98.17%  1.06    GiB
        Text:                           1.83%   20.32   MiB

Kernel Memory Map:                              7.19    GiB
        Size:                           13.76%  1013.16 MiB
        Free:                           86.24%  6.20    GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (THROTTLED)
        Storage pool Version:                   28
        Filesystem Version:                     5
        Memory Throttle Count:                  3.96k

ARC Misc:
        Deleted:                                1.68m
        Recycle Misses:                         72.33k
        Mutex Misses:                           7
        Evict Skips:                            7

ARC Size:                               20.00%  1.00    GiB
        Target Size: (Adaptive)         20.00%  1.00    GiB
        Min Size (Hard Limit):          20.00%  1.00    GiB
        Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.75%  960.06  MiB
        Frequently Used Cache Size:     6.25%   63.97   MiB

ARC Hash Breakdown:
        Elements Max:                           32.70k
        Elements Current:               71.77%  23.46k
        Collisions:                             302.87k
        Chain Max:                              5
        Chains:                                 2.26k
                                                                Page:  2
------------------------------------------------------------------------

ARC Efficiency:                                 2.41m
        Cache Hit Ratio:                93.64%  2.25m
        Cache Miss Ratio:               6.36%   152.99k
        Actual Hit Ratio:               81.53%  1.96m

        Data Demand Efficiency:         99.98%  1.19m
        Data Prefetch Efficiency:       61.05%  42.68k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             12.70%  286.05k
          Most Recently Used:           30.11%  678.21k
          Most Frequently Used:         56.96%  1.28m
          Most Recently Used Ghost:     0.09%   2.02k
          Most Frequently Used Ghost:   0.14%   3.13k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  52.67%  1.19m
          Prefetch Data:                1.16%   26.06k
          Demand Metadata:              34.10%  768.23k
          Prefetch Metadata:            12.07%  271.92k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.12%   184
          Prefetch Data:                10.87%  16.62k
          Demand Metadata:              83.51%  127.77k
          Prefetch Metadata:            5.50%   8.42k
                                                                Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 55.20m
        Hit Ratio:                      98.80%  54.53m
        Miss Ratio:                     1.20%   664.34k

        Colinear:                               664.34k
          Hit Ratio:                    0.03%   196
          Miss Ratio:                   99.97%  664.14k

        Stride:                                 54.49m
          Hit Ratio:                    100.00% 54.49m
          Miss Ratio:                   0.00%   114

DMU Misc:
        Reclaim:                                664.14k
          Successes:                    0.17%   1.12k
          Failures:                     99.83%  663.03k

        Streams:                                41.08k
          +Resets:                      0.10%   43
          -Resets:                      99.90%  41.04k
          Bogus:                                0
                                                                Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            8305745920
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            981729280
        vfs.zfs.mfu_ghost_metadata_lsize        23986688
        vfs.zfs.mfu_ghost_size                  1005715968
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              9241600
        vfs.zfs.mfu_size                        11002880
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        67137536
        vfs.zfs.mru_ghost_size                  67137536
        vfs.zfs.mru_data_lsize                  932244480
        vfs.zfs.mru_metadata_lsize              9625600
        vfs.zfs.mru_size                        965317632
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       41498112
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  1342177280
        vfs.zfs.arc_meta_used                   100359352
        vfs.zfs.arc_min                         1073741824
        vfs.zfs.arc_max                         5368709120
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            25698410496
        vfs.zfs.write_limit_max                 1070767104
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
                                                                Page:  7
------------------------------------------------------------------------
```


Note that
	
	



```
Kernel Memory:                                  1.08    GiB
        Data:                           98.17%  1.06    GiB
        Text:                           1.83%   20.32   MiB
```

Kept pretty much static on those values, it was only in the very beginning that I've had such high values

```
Kernel Memory:                                  1.86    GiB
        Data:                           98.94%  1.84    GiB
        Text:                           1.06%   20.22   MiB
```


----------



## Leander (May 1, 2012)

Btw. I have only one RAIDz1 running _(currently resilvering for testing purposes)_:
... The md[n] devices are only for temporary testing. But sill, each of them has its own source file on a seperate HDD.


```
zpool status -v
  pool: zStar
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Tue May  1 20:54:40 2012
    68,0G scanned out of 214G at 33,5M/s, 1h14m to go
    7,56G resilvered, 31,75% done
config:

        NAME                       STATE     READ WRITE CKSUM
        zStar                      DEGRADED     0     0     0
          raidz1-0                 DEGRADED     0     0     0
            md3                    ONLINE       0     0     0
            md4                    ONLINE       0     0     0
            md5                    ONLINE       0     0     0
            md6                    ONLINE       0     0     0
            md7                    ONLINE       0     0     0
            md8                    ONLINE       0     0     0
            replacing-5            UNAVAIL      0     0     0
              6048967463963296996  UNAVAIL      0     0     0  was /dev/md8/old
              md8                  ONLINE       0     0     0  (resilvering)
            md9                    ONLINE       0     0     0
            ada3p1                 ONLINE       0     0     0
            ada8p1                 ONLINE       0     0     0

errors: No known data errors
```


Thanks for hints


----------



## kpa (May 1, 2012)

Start with a very conservative setting of vfs.zfs.arc_max, let's say 1GB and up it in 1GB increments. Leave other settings to their defaults. This way you can very quickly tell if the crash is really from memory exhaustion caused by the ARC cache or something else.


----------



## gkontos (May 1, 2012)

Leander said:
			
		

> Anyway, after some tests I figured to get kernel panics under control by decreasing vfs.zfs.arc_max to 4G *(50% of my RAM)* and at the same time increase vfs.zfs.arc_min to 1G *(12.5% of my RAM)*.



Start by limiting the size of ARC_MAX and disabling PREFETCH. That should be enough.


```
vfs.zfs.prefetch_disable=1
vfs.zfs.arc_max="4096M"
```

Don't tweak any other values before you really know what you are doing.

Also please post the output of:

[CMD=""]# zfs get all zStar[/CMD]
[CMD=""]# zdb -DD zStar[/CMD]

Regards


----------



## usdmatt (May 1, 2012)

Not going to get involved in zfs tuning discussion as I have no expertise in it yet but just noting that there is a port of arc_summary.pl under sysutils/zfs-stats.


----------



## t1066 (May 2, 2012)

How do you set up your system? Where doea\s root reside and where do you put the swap file?


----------



## gkontos (May 2, 2012)

usdmatt said:
			
		

> Not going to get involved in zfs tuning discussion as I have no expertise in it yet but just noting that there is a port of arc_summary.pl under sysutils/zfs-stats.



I think that the statistics he posted were coming from that script.


----------



## usdmatt (May 3, 2012)

gkontos said:
			
		

> I think that the statistics he posted were coming from that script.



Yes, I know. He is using arc_summary.pl downloaded via the FreeBSD wiki, as mentioned in his post. I was just making people aware that there is a port of this now, so rather that finding it on the net you can just type:

[cmd=]pkg_add -r zfs-stats[/cmd]


----------



## kpa (May 4, 2012)

Please note that what I wrote above about the kmem tunables only applies to amd64, i386 is quite different because of limited address space and usually requires tuning of kmem settings for stable operation of ZFS.


----------

