# vm.kmem_size_max: 329853485875



## yayix (Aug 15, 2012)

On a server with 512GB RAM it appears that vm.kmem_size_max is not being auto-tuned to use >*329853485875* (~307GB).

On this machine vm.kmem_size is equal to vm.kmem_size_max

```
vm.kmem_size_max: 329853485875
vm.kmem_size: 329853485875
```

On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max to 330GB and vm.kmem_size automatically adjusts to 1GB even if I manually set it in /boot/loader.conf.

But on the machine with 512GB of RAM it just resets. For the machine to boot, we need to go to the loader prompt and issue:


```
OK set vm.kmem_size_max="300G"
OK boot
```

On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have vm.kmem_size_max is always set to *329853485875*. 

How can I increase vm.kmem_size_max to use at least 500GB? And how is *329853485875* determined (formula)? I need to increase vm.kmem_size_max and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say 490GB.

TIA!


----------



## yayix (Aug 15, 2012)

yayix said:
			
		

> But on the machine with 512GB of RAM it just resets. For the machine to boot, we need to go to the loader prompt and issue:
> 
> 
> ```
> ...



Sorry, forgot to state that the machine resets during boot because I have set this: 

/boot/loader.conf

```
vm.kmem_size_max="500G"
```


----------



## yayix (Aug 15, 2012)

*zfs-stats -a (part 1)*

[CMD=""]zfs-stats -a[/CMD]

```
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Aug 15 09:08:28 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root
 9:08AM  up 15:29, 2 users, load averages: 1.89, 1.75, 1.67

------------------------------------------------------------------------

System Memory:

        0.06%   292.86  MiB Active,     0.02%   111.19  MiB Inact
        47.21%  234.50  GiB Wired,      0.00%   896.00  KiB Cache
        51.07%  253.66  GiB Free,       1.65%   8.18    GiB Gap

        Real Installed:                         512.00  GiB
        Real Available:                 99.96%  511.82  GiB
        Real Managed:                   97.05%  496.73  GiB

        Logical Total:                          512.00  GiB
        Logical Used:                   50.43%  258.23  GiB
        Logical Free:                   49.57%  253.77  GiB

Kernel Memory:                                  227.74  GiB
        Data:                           99.99%  227.72  GiB
        Text:                           0.01%   21.64   MiB

Kernel Memory Map:                              304.23  GiB
        Size:                           74.81%  227.60  GiB
        Free:                           25.19%  76.63   GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                28.21m
        Recycle Misses:                         39.49k
        Mutex Misses:                           366.36k
        Evict Skips:                            677.56k

ARC Size:                               74.47%  228.03  GiB
        Target Size: (Adaptive)         74.47%  228.03  GiB
        Min Size (Hard Limit):          50.00%  153.10  GiB
        Max Size (High Water):          2:1     306.20  GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.48%  213.17  GiB
        Frequently Used Cache Size:     6.52%   14.86   GiB

ARC Hash Breakdown:
        Elements Max:                           5.64m
        Elements Current:               100.00% 5.64m
        Collisions:                             14.49m
        Chain Max:                              8
        Chains:                                 1.24m

------------------------------------------------------------------------

ARC Efficiency:                                 259.31m
        Cache Hit Ratio:                92.53%  239.93m
        Cache Miss Ratio:               7.47%   19.37m
        Actual Hit Ratio:               92.40%  239.59m

        Data Demand Efficiency:         99.97%  220.93m
        Data Prefetch Efficiency:       1.45%   19.40m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           28.30%  67.90m
          Most Frequently Used:         71.55%  171.68m
          Most Recently Used Ghost:     0.13%   315.89k
          Most Frequently Used Ghost:   0.28%   672.06k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  92.05%  220.85m
          Prefetch Data:                0.12%   282.17k
          Demand Metadata:              7.81%   18.74m
          Prefetch Metadata:            0.03%   62.46k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.37%   71.62k
          Prefetch Data:                98.70%  19.12m
          Demand Metadata:              0.88%   170.76k
          Prefetch Metadata:            0.04%   8.61k

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        1.90m
        Tried Lock Failures:                    272.32k
        IO In Progress:                         5
        Low Memory Aborts:                      0
        Free on Write:                          37.71k
        Writes While Full:                      30.41k
        R/W Clashes:                            140
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           461.17m

L2 ARC Size: (Adaptive)                         231.94  GiB
        Header Size:                    0.18%   416.63  MiB

L2 ARC Breakdown:                               19.37m
        Hit Ratio:                      0.40%   77.99k
        Miss Ratio:                     99.60%  19.29m
        Feeds:                                  80.72k

L2 ARC Buffer:
        Bytes Scanned:                          36.73   TiB
        Buffer Iterations:                      80.72k
        List Iterations:                        4.56m
        NULL List Iterations:                   391.60k

L2 ARC Writes:
        Writes Sent:                    100.00% 45.50k

------------------------------------------------------------------------
```


----------



## yayix (Aug 15, 2012)

*zfs-stats -a (part 2)*

[CMD=""]zfs-stats -a[/CMD]


```
File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 952.47m
        Hit Ratio:                      99.82%  950.71m
        Miss Ratio:                     0.18%   1.76m

        Colinear:                               1.76m
          Hit Ratio:                    0.01%   241
          Miss Ratio:                   99.99%  1.76m

        Stride:                                 929.86m
          Hit Ratio:                    100.00% 929.85m
          Miss Ratio:                   0.00%   2.24k

DMU Misc:
        Reclaim:                                1.76m
          Successes:                    0.63%   11.08k
          Failures:                     99.37%  1.75m

        Streams:                                20.86m
          +Resets:                      0.00%   440
          -Resets:                      100.00% 20.86m
          Bogus:                                0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            329853485875
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   242597267456
        vfs.zfs.mfu_ghost_data_lsize            64166322176
        vfs.zfs.mfu_ghost_metadata_lsize        645829632
        vfs.zfs.mfu_ghost_size                  64812151808
        vfs.zfs.mfu_data_lsize                  176958659584
        vfs.zfs.mfu_metadata_lsize              31165952
        vfs.zfs.mfu_size                        177167786496
        vfs.zfs.mru_ghost_data_lsize            174980341760
        vfs.zfs.mru_ghost_metadata_lsize        336025600
        vfs.zfs.mru_ghost_size                  175316367360
        vfs.zfs.mru_data_lsize                  65570037760
        vfs.zfs.mru_metadata_lsize              101715968
        vfs.zfs.mru_size                        65818363904
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       24707072
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  82194936012
        vfs.zfs.arc_meta_used                   2292602848
        vfs.zfs.arc_min                         164389872024
        vfs.zfs.arc_max                         328779744048
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            1648684572672
        vfs.zfs.write_limit_max                 68695190528
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               96
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------
```


----------



## gkontos (Aug 15, 2012)

yayix said:
			
		

> How can I increase vm.kmem_size_max to use at least 500GB? And how is *329853485875* determined (formula)? I need to increase vm.kmem_size_max and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say 490GB.
> 
> TIA!



Why would you want to do that?


----------



## yayix (Aug 15, 2012)

gkontos said:
			
		

> Why would you want to do that?



This machine will be mainly used for postgreSQL. We have a huge db (3TB and growing) and we need to have as much of it as we can on zfs' ARC. I don't think there's any reason to not want to utilize all RAM and leave just enough for other stuff. Moreso, on our 8.2 and 9 machines vm.kmem_size is always auto-tuned to almost the same size as our installed RAM. What I've tuned on those machines is lower vfs.zfs.arc_max to ~75% of vm.kmem_size and that have worked well for us and the machines does not swap out.

By the way, we were able to successfully bump vm.kmem_size_max and vm.kmem_size to 400GB. Tried 450GB and 500GB, server boots but it spewed out garbage during the boot up process.

[CMD=""]zfs-stats -a[/CMD]


```
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Aug 15 12:46:22 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root
12:46PM  up 25 mins, 2 users, load averages: 0.07, 0.03, 0.12

------------------------------------------------------------------------

System Memory:

        0.02%   94.02   MiB Active,     0.01%   56.74   MiB Inact
        1.00%   4.98    GiB Wired,      0.00%   792.00  KiB Cache
        98.85%  491.03  GiB Free,       0.12%   589.25  MiB Gap

        Real Installed:                         512.00  GiB
        Real Available:                 99.96%  511.82  GiB
        Real Managed:                   97.05%  496.73  GiB

        Logical Total:                          512.00  GiB
        Logical Used:                   4.08%   20.91   GiB
        Logical Free:                   95.92%  491.09  GiB

Kernel Memory:                                  1.72    GiB
        Data:                           98.77%  1.70    GiB
        Text:                           1.23%   21.64   MiB

Kernel Memory Map:                              400.00  GiB
        Size:                           0.42%   1.69    GiB
        Free:                           99.58%  398.31  GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                27
        Recycle Misses:                         0
        Mutex Misses:                           0
        Evict Skips:                            275

ARC Size:                               0.10%   427.15  MiB
        Target Size: (Adaptive)         100.00% 399.00  GiB
        Min Size (Hard Limit):          12.50%  49.88   GiB
        Max Size (High Water):          8:1     399.00  GiB

ARC Size Breakdown:
        Recently Used Cache Size:       50.00%  199.50  GiB
        Frequently Used Cache Size:     50.00%  199.50  GiB

ARC Hash Breakdown:
        Elements Max:                           4.05k
        Elements Current:               99.95%  4.05k
        Collisions:                             0
        Chain Max:                              0
        Chains:                                 0

------------------------------------------------------------------------

ARC Efficiency:                                 69.58k
        Cache Hit Ratio:                94.18%  65.53k
        Cache Miss Ratio:               5.82%   4.05k
        Actual Hit Ratio:               94.02%  65.42k

        Data Demand Efficiency:         99.87%  52.76k
        Data Prefetch Efficiency:       0.00%   3.29k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.17%   113
          Most Recently Used:           15.93%  10.44k
          Most Frequently Used:         83.89%  54.98k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  80.41%  52.69k
          Prefetch Data:                0.00%   0
          Demand Metadata:              19.41%  12.72k
          Prefetch Metadata:            0.18%   116

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  1.63%   66
          Prefetch Data:                81.20%  3.29k
          Demand Metadata:              12.23%  495
          Prefetch Metadata:            4.94%   200

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        0
        Tried Lock Failures:                    11
        IO In Progress:                         0
        Low Memory Aborts:                      0
        Free on Write:                          0
        Writes While Full:                      25
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           327.45k

L2 ARC Size: (Adaptive)                         417.07  MiB
        Header Size:                    0.00%   0

L2 ARC Breakdown:                               4.03k
        Hit Ratio:                      0.00%   0
        Miss Ratio:                     100.00% 4.03k
        Feeds:                                  1.48k

L2 ARC Buffer:
        Bytes Scanned:                          29.98   GiB
        Buffer Iterations:                      1.48k
        List Iterations:                        94.33k
        NULL List Iterations:                   44.80k

L2 ARC Writes:
        Writes Sent:                    100.00% 47
```

continued on next post..


----------



## yayix (Aug 15, 2012)

*continuation zfs-stats -a with vm.kmem_size_max and vm.kmem_size set to "400G"*

Continued from previous post. With vm.kmem_size_max and vm.kmem_size set to "400G".


```
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 121.02k
        Hit Ratio:                      99.67%  120.61k
        Miss Ratio:                     0.33%   404

        Colinear:                               404
          Hit Ratio:                    0.25%   1
          Miss Ratio:                   99.75%  403

        Stride:                                 117.26k
          Hit Ratio:                    100.00% 117.26k
          Miss Ratio:                   0.00%   0

DMU Misc:
        Reclaim:                                403
          Successes:                    4.71%   19
          Failures:                     95.29%  384

        Streams:                                3.36k
          +Resets:                      0.09%   3
          -Resets:                      99.91%  3.36k
          Bogus:                                0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            429496729600
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        429496729600
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            9216
        vfs.zfs.mfu_ghost_metadata_lsize        0
        vfs.zfs.mfu_ghost_size                  9216
        vfs.zfs.mfu_data_lsize                  424943104
        vfs.zfs.mfu_metadata_lsize              365568
        vfs.zfs.mfu_size                        425761280
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        114688
        vfs.zfs.mru_ghost_size                  114688
        vfs.zfs.mru_data_lsize                  11144704
        vfs.zfs.mru_metadata_lsize              4005376
        vfs.zfs.mru_size                        18676736
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       101888
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  107105746944
        vfs.zfs.arc_meta_used                   11810416
        vfs.zfs.arc_min                         53552873472
        vfs.zfs.arc_max                         428422987776
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            1648684572672
        vfs.zfs.write_limit_max                 68695190528
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               96
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------
```


----------



## gkontos (Aug 15, 2012)

I don't think that at this stage you need to change any of those values. 
Your memory capacity is much higher for any type of ZFS tuning regarding arc and vm.kmem_size.


----------



## yayix (Aug 15, 2012)

gkontos said:
			
		

> I don't think that at this stage you need to change any of those values.
> Your memory capacity is much higher for any type of ZFS tuning regarding arc and vm.kmem_size.



Thanks!

Exactly why it needs to be tuned for our needs. Since FreeBSD imposes a limit on vm.kmem_size_max and have probably hard-coded it to ~307GB, we are blindly doing a trial and error on this setting. The machine is not yet in production but will be soon and once it is we won't have the luxury of doing a reboot for tuning as rebooting takes a while (~20mins).

Again bringing up the question of how 329853485875 (307GB) is computed? And what more if the installed RAM is 1TB?

From: Frequently Asked Questions for FreeBSD 7.X, 8.X, and 9.X [4.2 Memory]


> 4.2.1. Does FreeBSD support more than 4 GB of memory (RAM)? More than 16 GB? More than 48 GB?
> 
> Yes. FreeBSD as an operating system generally supports as much physical memory (RAM) as the platform it is running on does. Keep in mind that different platforms have different limits for memory; for example i386â„¢ without PAE supports at most 4 GB of memory (and usually less than that because of PCI address space) and i386 with PAE supports at most 64 GB memory. AMD64 platforms currently deployed support up to 1 TB of physical memory.



Hope there are others who can shed some light on this?


----------



## yayix (Aug 19, 2012)

Hey guys,

The topic has not yet been resolved but Alan Cox _[presuming that's what "alc" means -- mod.]_ has some input and is looking into it. Here's the mailing list thread if you are interested: http://lists.freebsd.org/pipermail/freebsd-performance/2012-August/subject.html#4631


----------



## michaeladm (Nov 26, 2013)

I have this problem too, but if 
	
	



```
vfs.zfs.vdev.cache.size=16M
```
 or more M - freebsd FreeBSD doesn't panic any more. (FreeBSD 9.2-STABLE + ZFS zpool)


----------

