# FreeBSD 9 64bit with ZFS crashes every 3 days



## belon_cfy (Feb 23, 2012)

*Problem solved. *
http://forums.freebsd.org/showpost.php?p=181397&postcount=52

-------------------------------------------------------------------------------
Hi.

I*'*m using FreeBSD 9 64bit with ZFS as a backup server, only NFS service is running, however it keep*s* crashing every 3 days after. 

The server hardware is Intel Core I3-2100, 4GB Ram and 4 X 2TB Seagate HDD. The server was running fine previously with ESXI 5.0 for more than three months without a single crash.

I suspected the problem is due to insufficient memory for arc so *I* have reduced the arc_max to 2GB only, reserved another 2GB for others.

```
vfs.zfs.arc_max="2G"
```

Below is my memory status captured every one minute. I notice that the server crashed after my free memory fell below 32MB.

```
Thu Feb 23 00:28:01 2012 Mem: 3768K Active, 448K Inact, 3730M Wired, 10M Cache, 92M Free
Thu Feb 23 00:28:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:29:03 2012 Mem: 5388K Active, 140K Inact, 3744M Wired, 9284K Cache, 78M Free
Thu Feb 23 00:29:03 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:30:03 2012 Mem: 3432K Active, 584K Inact, 3745M Wired, 10M Cache, 77M Free
Thu Feb 23 00:30:03 2012 Swap: 4096M Total, 27M Used, 4068M Free
Thu Feb 23 00:31:06 2012 Mem: 5060K Active, 708K Inact, 3744M Wired, 9276K Cache, 78M Free
Thu Feb 23 00:31:06 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:32:01 2012 Mem: 6376K Active, 584K Inact, 3753M Wired, 8188K Cache, 68M Free
Thu Feb 23 00:32:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:33:01 2012 Mem: 3356K Active, 80K Inact, 3754M Wired, 11M Cache, 68M Free
Thu Feb 23 00:33:01 2012 Swap: 4096M Total, 28M Used, 4068M Free
Thu Feb 23 00:34:00 2012 Mem: 3692K Active, 712K Inact, 3739M Wired, 10M Cache, 83M Free
Thu Feb 23 00:34:00 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:35:01 2012 Mem: 4140K Active, 216K Inact, 3767M Wired, 10M Cache, 55M Free
Thu Feb 23 00:35:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:36:01 2012 Mem: 2312K Active, 156K Inact, 3768M Wired, 7180K Cache, 59M Free
Thu Feb 23 00:36:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:37:00 2012 Mem: 3096K Active, 72K Inact, 3749M Wired, 7484K Cache, 77M Free
Thu Feb 23 00:37:00 2012 Swap: 4096M Total, 28M Used, 4068M Free
Thu Feb 23 00:38:00 2012 Mem: 2816K Active, 508K Inact, 3762M Wired, 6948K Cache, 64M Free
Thu Feb 23 00:38:00 2012 Swap: 4096M Total, 28M Used, 4068M Free
Thu Feb 23 00:39:01 2012 Mem: 4044K Active, 220K Inact, 3750M Wired, 6380K Cache, 76M Free
Thu Feb 23 00:39:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:40:04 2012 Mem: 3416K Active, 224K Inact, 3756M Wired, 7004K Cache, 70M Free
Thu Feb 23 00:40:04 2012 Swap: 4096M Total, 28M Used, 4068M Free
Thu Feb 23 00:41:03 2012 Mem: 5084K Active, 332K Inact, 3758M Wired, 5740K Cache, 67M Free
Thu Feb 23 00:41:03 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:42:01 2012 Mem: 3904K Active, 184K Inact, 3772M Wired, 6724K Cache, 54M Free
Thu Feb 23 00:42:01 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:43:01 2012 Mem: 3696K Active, 140K Inact, 3761M Wired, 6968K Cache, 65M Free
Thu Feb 23 00:43:01 2012 Swap: 4096M Total, 27M Used, 4068M Free
Thu Feb 23 00:44:01 2012 Mem: 4084K Active, 572K Inact, 3777M Wired, 5268K Cache, 49M Free
Thu Feb 23 00:44:01 2012 Swap: 4096M Total, 27M Used, 4068M Free
Thu Feb 23 00:45:05 2012 Mem: 5332K Active, 408K Inact, 3792M Wired, 2368K Cache, 36M Free
Thu Feb 23 00:45:05 2012 Swap: 4096M Total, 27M Used, 4069M Free
Thu Feb 23 00:46:11 2012 Mem: 5108K Active, 692K Inact, 3797M Wired, 1876K Cache, 32M Free
Thu Feb 23 00:46:11 2012 Swap: 4096M Total, 26M Used, 4070M Free
```

May *I* know whether increasing kmem_size will help to avoid server down due to memory exhaustion? Below is my current setting.

```
vm.kmem_size: 4023611392
```

I would like to increase it to 6GB and try again. 

Any comment?


----------



## phoenix (Feb 23, 2012)

Are you using dedupe?

What's your pool config? 2x mirror vdevs? 1x raid1 vdev? Any cache vdevs?

Don't mess with kmem settings. kmem_size just shows the current size. kmem_size_max shows the full address space, which will be something like 64GB.

What is running your backups? What other software is running on there?

Something is running out of control and Wiring down all your RAM, leaving nothing for the rest of OS and starving the system leading to a lockup.  Most likely is that your ARC is running out of control. Usual culprit is using dedupe without enough RAM.


----------



## belon_cfy (Feb 23, 2012)

Nope, *I'm* using compression instead of dedupe none of the volume with dedupe enabled.

Forgot to attach my raid status.

```
NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
```

No software running on the server, this is only a nfs server sharing folder to others for rsync backup only. 

Forgot to mention that, the server first crashed at 1:10am, and the second crashed at 00:46, possible any cronjob causing it?


----------



## idownes (Feb 23, 2012)

I'm seeing ZFS arc_*meta*_used taking all RAM.

Try logging the ZFS sysctl as I've suggested here http://forums.freebsd.org/showpost.php?p=167623&postcount=12 and check out vfs.zfs.arc_meta_used compared to vfs.zfs.arc_meta_limit


----------



## belon_cfy (Feb 23, 2012)

Hi idownes,

Seem*s* *I'm* having the exactly same issue as you. My server is keeping more than 3 TB of small files, periodic 100.chksetuid took my I/O for checking so *I* disabled it. 

By the way, below are my zfs sysctl values, seem*s* the vfs.zfs.arc_meta_used: 2944727520 already exceed*s* the vfs.zfs.arc_meta_limit: 737467392.

Any idea?



```
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_lsize: 6951424
vfs.zfs.mfu_ghost_metadata_lsize: 2373072896
vfs.zfs.mfu_ghost_size: 2380024320
vfs.zfs.mfu_data_lsize: 4774912
vfs.zfs.mfu_metadata_lsize: 65536
vfs.zfs.mfu_size: 5227520
vfs.zfs.mru_ghost_data_lsize: 0
vfs.zfs.mru_ghost_metadata_lsize: 99926016
vfs.zfs.mru_ghost_size: 99926016
vfs.zfs.mru_data_lsize: 395264
vfs.zfs.mru_metadata_lsize: 2845324288
vfs.zfs.mru_size: 2851828736
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 65536
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 737467392
vfs.zfs.arc_meta_used: 2944727520
vfs.zfs.arc_min: 368733696
vfs.zfs.arc_max: 2949869568
vfs.zfs.dedup.prefetch: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.write_limit_override: 0
vfs.zfs.write_limit_inflated: 12544475136
vfs.zfs.write_limit_max: 522686464
vfs.zfs.write_limit_min: 33554432
vfs.zfs.write_limit_shift: 3
vfs.zfs.no_write_throttle: 0
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.mg_alloc_failures: 8
vfs.zfs.check_hostid: 1
vfs.zfs.recover: 0
vfs.zfs.txg.synctime_ms: 1000
vfs.zfs.txg.timeout: 5
vfs.zfs.scrub_limit: 10
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 6
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 10
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.zio.use_uma: 0
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 28
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
```


----------



## phoenix (Feb 23, 2012)

It may be periodic(8) scripts running at night scanning all files looking for various things.

One of the things I've had to modify on my ZFS backups servers is /etc/locate.rc, mainly to remove paths from the database (don't need all the files being backed up in my locate db):

```
PRUNEPATH="/tmp /usr/tmp /var/tmp /var/db/portsnap /backups /cameras
PRUNEDIRS=".zfs"
```

I've also modified /etc/periodic.conf to disable a bunch of checks I don't need, mainly dealing with sendmail, named, rwho, setuid, ipfilter.  See /etc/defaults/periodic.conf for what can be added to /etc/periodic.conf.


----------



## belon_cfy (Feb 23, 2012)

If the vfs.zfs.arc_meta_used already exceed*s* the vfs.zfs.arc_meta_limit, what should *I* do to cap the arc_meta_used below the arc_meta_limit?


----------



## belon_cfy (Feb 23, 2012)

By the way, this is my current vfs.zfs.arc_meta values. Why can the meta_used exceed the meta_limit in such a case? Is this due to the bug in zfs or FreeBSD?


```
Thu Feb 23 13:01:20 2012 vfs.zfs.arc_meta_limit: 737467392
Thu Feb 23 13:01:20 2012 vfs.zfs.arc_meta_used: 2606797560
```


----------



## belon_cfy (Feb 23, 2012)

Forgot to mention, the vfs.zfs.arc_meta_used keep going rapidly when doing disk scrub.


----------



## idownes (Feb 23, 2012)

I asked this question a few days ago (here and freebsd-fs@) but haven't yet received any answers. I assume that it's a bug.

http://forums.freebsd.org/showthread.php?t=29969

Also, look at these related posts for more information and workarounds

http://forums.freebsd.org/showthread.php?t=29994


----------



## belon_cfy (Feb 23, 2012)

Seems the arc_meta_limit is a soft limit instead of a hard limit and *I* believe there is a bug lead to arc_meta_used exceed the limit when scrubbing zfs. 

I'm copying a small file to the server and the arc_meta_used never exceeds the limit for more than 10% 

```
vfs.zfs.arc_meta_limit: 737467392
vfs.zfs.arc_meta_used: 736363616
```

But when scrubbing, it can be easily exceeded up to 5 times the limit, seems the soft limit didn't take effect in this operation.


----------



## DutchDaemon (Feb 23, 2012)

@belon_cfy Start formatting your posts correctly now, thanks.


----------



## belon_cfy (Feb 26, 2012)

DutchDaemon , thanks for reminding.

Another FreeBSD server crashed again yesterday. Seems it was due to the memory exhaustion again but I still can't figure out which process keeps consuming my available memory. 

My server specification as below:

```
Intel Xeon 5335 Quadcore 
4GB ECC Ram
4 X 2TB HDD with RAID10
no dedupe is enabled, only compression
```


```
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Mem: 6980K Active, 460K Inact, 3906M Wired, 18M Free
```


Below are the vfs.zfs values I have logged before it crashed.

```
Sat Feb 25 19:42:40 2012 vfs.zfs.l2c_only_size: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_ghost_data_lsize: 2856960
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_ghost_metadata_lsize: 88188928
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_ghost_size: 91045888
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_data_lsize: 399360
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_metadata_lsize: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.mfu_size: 12640256
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_ghost_data_lsize: 2710016
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_ghost_metadata_lsize: 234015232
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_ghost_size: 236725248
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_data_lsize: 624788480
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_metadata_lsize: 83579392
Sat Feb 25 19:42:40 2012 vfs.zfs.mru_size: 835642880
Sat Feb 25 19:42:40 2012 vfs.zfs.anon_data_lsize: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.anon_metadata_lsize: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.anon_size: 12377088
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_norw: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_feed_again: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_noprefetch: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_feed_min_ms: 200
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_feed_secs: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_headroom: 2
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_write_boost: 8388608
Sat Feb 25 19:42:40 2012 vfs.zfs.l2arc_write_max: 8388608
Sat Feb 25 19:42:40 2012 vfs.zfs.arc_meta_limit: 536870912
Sat Feb 25 19:42:40 2012 vfs.zfs.arc_meta_used: 438924456
Sat Feb 25 19:42:40 2012 vfs.zfs.arc_min: 268435456
Sat Feb 25 19:42:40 2012 vfs.zfs.arc_max: 2147483648
Sat Feb 25 19:42:40 2012 vfs.zfs.dedup.prefetch: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.mdcomp_disable: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.write_limit_override: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.write_limit_inflated: 12818632704
Sat Feb 25 19:42:40 2012 vfs.zfs.write_limit_max: 534109696
Sat Feb 25 19:42:40 2012 vfs.zfs.write_limit_min: 33554432
Sat Feb 25 19:42:40 2012 vfs.zfs.write_limit_shift: 3
Sat Feb 25 19:42:40 2012 vfs.zfs.no_write_throttle: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.zfetch.array_rd_sz: 1048576
Sat Feb 25 19:42:40 2012 vfs.zfs.zfetch.block_cap: 256
Sat Feb 25 19:42:40 2012 vfs.zfs.zfetch.min_sec_reap: 2
Sat Feb 25 19:42:40 2012 vfs.zfs.zfetch.max_streams: 8
Sat Feb 25 19:42:40 2012 vfs.zfs.prefetch_disable: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.mg_alloc_failures: 8
Sat Feb 25 19:42:40 2012 vfs.zfs.check_hostid: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.recover: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.txg.synctime_ms: 1000
Sat Feb 25 19:42:40 2012 vfs.zfs.txg.timeout: 5
Sat Feb 25 19:42:40 2012 vfs.zfs.scrub_limit: 10
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.cache.bshift: 16
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.cache.size: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.cache.max: 16384
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.write_gap_limit: 4096
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.read_gap_limit: 32768
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.aggregation_limit: 131072
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.ramp_rate: 2
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.time_shift: 6
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.min_pending: 4
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.max_pending: 10
Sat Feb 25 19:42:40 2012 vfs.zfs.vdev.bio_flush_disable: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.cache_flush_disable: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.zil_replay_disable: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.zio.use_uma: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.version.zpl: 5
Sat Feb 25 19:42:40 2012 vfs.zfs.version.spa: 28
Sat Feb 25 19:42:40 2012 vfs.zfs.version.acl: 1
Sat Feb 25 19:42:40 2012 vfs.zfs.debug: 0
Sat Feb 25 19:42:40 2012 vfs.zfs.super_owner: 0
```


----------



## belon_cfy (Mar 4, 2012)

2 servers crashed again last night after 3 days. Both servers contain a lot of small files and occupied 3TB space per server.

Changing the kmem and kmem_max to 3.5GB doesn't help to keep my servers stable. 

I'm trying to change the arc_max to 512MB on the first server and remain the default setting on the other. Will report whether both servers are able to survive for more than a week. 

Is it possible caused by the compressed zfs swap volume? Theoretically compression might consumes some memory during swap in/out and it might crashes on memory exhausted server. Correct me if I'm wrong.


----------



## kpa (Mar 4, 2012)

Leave the kmem* tunables alone on AMD64 unless you have a REALLY good reason to change them, starting with 8.2-RELEASE there's generally no reason to touch them on AMD64. Tuning of vfs.zfs.arc_max is recommended however and can be really helpful to limit the size of ARC to a sensible size.


----------



## belon_cfy (Mar 4, 2012)

kpa said:
			
		

> Leave the kmem* tunables alone on AMD64 unless you have a REALLY good reason to change them, starting with 8.2-RELEASE there's generally no reason to touch them on AMD64. Tuning of vfs.zfs.arc_max is recommended however and can be really helpful to limit the size of ARC to a sensible size.


It is what I did before, I changed the arc_max to 2GB on 4GB server and it did make the server run longer but crashed again eventually due to the same reason.

I have changed it to 512MB, will it still utilize my free memory as read cache?


----------



## olav (Mar 5, 2012)

You sure this is not a hardware related issue? Have you checked logs for disks dropping out because of heavy load?

I don't do any memory tunings for ZFS. It's not necessary anymore.


----------



## phoenix (Mar 5, 2012)

belon_cfy said:
			
		

> 2 servers crashed again last night after 3 days. Both servers contain a lot of small files and occupied 3TB space per server.
> 
> Changing the kmem and kmem_max to 3.5GB doesn't help to keep my servers stable.
> 
> ...



Don't use swap on zfs. Bad things happen, as you have noticed.  You need RAM for ARC. You need ARC to track pool usage, which includes your swap volume. When you run out of RAM, you send things to swap ... which leads to more ARC usage, which leads to less RAM, which means you need to send more stuff to swap, and the cycle contnues until BOOM.


----------



## belon_cfy (Mar 5, 2012)

olav said:
			
		

> You sure this is not a hardware related issue? Have you checked logs for disks dropping out because of heavy load?
> 
> I don't do any memory tunings for ZFS. It's not necessary anymore.


I don't think it is related to the hardware issue because I'm having completely the same symptom on 2 different servers(Xeon and Core i3 2100 with 4GB Ram). Both servers crashed almost at the same period. 

Both servers are able to survive for more than 3-4 months previously as ESXI host with *NexentaStor* installed.


----------



## belon_cfy (Mar 5, 2012)

phoenix said:
			
		

> Don't use swap on zfs. Bad things happen, as you have noticed.  You need RAM for ARC. You need ARC to track pool usage, which includes your swap volume. When you run out of RAM, you send things to swap ... which leads to more ARC usage, which leads to less RAM, which means you need to send more stuff to swap, and the cycle contnues until BOOM.



Hi Phoenix, 
Thanks for pointing out. The swap is running on ZFS since the root partition is ZFS too. 

I hope to retain the swap volume on zfs since there is no available space for me to create a new swap partition. Will try to disable the primarycache and secondarycache on swap volume and see how is everything going on.


----------



## phoenix (Mar 5, 2012)

If you need swap space, then find a USB stick somewhere, plug it in, and configure that as your swap space.  You really need to remove the swap-on-ZVol if you want your server to stop crashing.


----------



## belon_cfy (Mar 8, 2012)

The problem is happening again, something is consuming all of my memory and causing heavy swap activity.

I have changed the arc_max to 1G on a 4GB server
Any idea?


```
48 processes:  1 running, 47 sleeping
CPU:  0.0% user,  0.0% nice,  3.1% system,  0.4% interrupt, 96.5% idle
Mem: 68K Active, 32K Inact, 3751M Wired, 12M Cache, 73M Free
Swap: 8192M Total, 32M Used, 8160M Free, 1012K Out

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
39788 root       32  20    0 10052K   512K rpcsvc  1   0:05  0.49% nfsd
16722 root        1  20    0 18404K    16K nanslp  1  21:23  0.00% vmstat
 1518 root        1  20    0 35604K    72K nanslp  0   2:12  0.00% zpool
 1444 root        1  20    0 68016K    32K select  1   0:48  0.00% sshd
 1458 root        1  24    0 19964K    16K nanslp  0   0:25  0.00% perl5.12.4
 1436 root        1  20    0 68016K    32K select  1   0:17  0.00% sshd
 1464 root        1  20    0 68016K    48K select  1   0:16  0.00% sshd
 6718 root        1  20    0 68016K    32K select  0   0:14  0.00% sshd
 1414 root        1  20    0 68016K    32K select  2   0:09  0.00% sshd
 1324 root        1  20    0 20384K    32K select  0   0:02  0.00% sendmail
 1334 root        1  20    0 14260K     0K nanslp  1   0:02  0.00% <cron>
 1035 root        1  20    0 14264K    16K select  1   0:01  0.00% rpcbind
 1012 root        1  20    0 12184K    32K select  2   0:01  0.00% syslogd
 1408 root        1  20    0 68016K    32K select  3   0:01  0.00% sshd
 1190 root        1  52    0 14264K    16K rpcsvc  2   0:00  0.00% rpc.lockd
39866 root        1  20    0 16700K   440K CPU0    0   0:00  0.00% top
 1184 root        1  20    0   268M    32K select  3   0:00  0.00% rpc.statd
 1162 root        1  20    0 10052K    16K select  2   0:00  0.00% nfsuserd
 1163 root        1  20    0 10052K    16K select  2   0:00  0.00% nfsuserd
 1161 root        1  20    0 10052K    16K select  1   0:00  0.00% nfsuserd
 1160 root        1  20    0 10052K    16K select  2   0:00  0.00% nfsuserd
 1417 root        1  20    0 17664K    32K ttyin   0   0:00  0.00% csh
 1328 smmsp       1  20    0 20384K     0K pause   0   0:00  0.00% <sendmail>
 1439 root        1  20    0 17664K     0K pause   2   0:00  0.00% <csh>
 1175 root        1  20    0 12180K    16K select  0   0:00  0.00% mountd
39787 root        1  20    0 10052K    16K select  2   0:00  0.00% nfsd
 6732 root        1  20    0 17664K     0K pause   1   0:00  0.00% <csh>
 1454 root        1  20    0 17664K     0K pause   0   0:00  0.00% <csh>
 1474 root        1  20    0 17664K     0K pause   0   0:00  0.00% <csh>
39879 root        1  21    0 19964K     0K nanslp  1   0:00  0.00% <perl5.12.4>
39881 root        1  23    0 19964K     0K nanslp  3   0:00  0.00% <perl5.12.4>
39880 root        1  20    0 19964K     0K nanslp  3   0:00  0.00% <perl5.12.4>
 1411 root        1  24    0 17664K     0K pause   0   0:00  0.00% <csh>
```


----------



## phoenix (Mar 8, 2012)

Uhm, *32 MB* of swap use is not "heavy swap usage".


----------



## peetaur (Mar 8, 2012)

Please post output from:

`# uname -a`
(thought someone else would have asked already)

and

`# zfs-stats -a`


And BTW, I found this to be very untrue on my system:


			
				kpa said:
			
		

> Leave the kmem* tunables alone on AMD64 unless you have a REALLY good reason to change them, starting with 8.2-RELEASE there's generally no reason to touch them on AMD64. Tuning of vfs.zfs.arc_max is recommended however and can be really helpful to limit the size of ARC to a sensible size.



With 8.2-RELEASE and STABLE from September 2011, my 48GB RAM zfs on root system wouldn't even use more than a few gigs of memory until I set vm.kmem_size. And another reason is my dual 10Gbps network card wouldn't work without setting it, because it wanted much more memory. Now it eats up the RAM nicely, saving just enough to share "Mem: 8948K Active, 43M Inact, 42G Wired, 23M Cache, 4730M Free".

If you rephrase that to say "starting with 8-STABLE from <fill in a date after September 2011>", then I would need to test it again, but otherwise I am sure that for my system, the above is not always correct. (but I am not so sure that I would make a bet that if I removed the kmem setting that it would use <4GB again, since it is strange and I have not yet performed that particular experiment)


----------



## belon_cfy (Mar 9, 2012)

Hi peetaur
below is my FreeBSD detail.

```
FreeBSD storage20 9.0-RELEASE FreeBSD 9.0-RELEASE amd64
```

My zfs parameters during that time.

```
Fri Mar  9 00:00:41 2012 vfs.zfs.l2c_only_size: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_ghost_data_lsize: 6148096
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_ghost_metadata_lsize: 55703552
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_ghost_size: 61851648
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_data_lsize: 39424
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_metadata_lsize: 1097728
Fri Mar  9 00:00:41 2012 vfs.zfs.mfu_size: 10587136
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_ghost_data_lsize: 13315072
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_ghost_metadata_lsize: 75742208
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_ghost_size: 89057280
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_data_lsize: 17732608
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_metadata_lsize: 10712576
Fri Mar  9 00:00:41 2012 vfs.zfs.mru_size: 79090176
Fri Mar  9 00:00:41 2012 vfs.zfs.anon_data_lsize: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.anon_metadata_lsize: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.anon_size: 231936
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_norw: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_feed_again: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_noprefetch: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_feed_min_ms: 200
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_feed_secs: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_headroom: 2
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_write_boost: 8388608
Fri Mar  9 00:00:41 2012 vfs.zfs.l2arc_write_max: 8388608
Fri Mar  9 00:00:41 2012 vfs.zfs.arc_meta_limit: 268435456
Fri Mar  9 00:00:41 2012 vfs.zfs.arc_meta_used: 141534312
Fri Mar  9 00:00:41 2012 vfs.zfs.arc_min: 134217728
Fri Mar  9 00:00:41 2012 vfs.zfs.arc_max: 1073741824
Fri Mar  9 00:00:41 2012 vfs.zfs.dedup.prefetch: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.mdcomp_disable: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.write_limit_override: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.write_limit_inflated: 12544475136
Fri Mar  9 00:00:41 2012 vfs.zfs.write_limit_max: 522686464
Fri Mar  9 00:00:41 2012 vfs.zfs.write_limit_min: 33554432
Fri Mar  9 00:00:41 2012 vfs.zfs.write_limit_shift: 3
Fri Mar  9 00:00:41 2012 vfs.zfs.no_write_throttle: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.zfetch.array_rd_sz: 1048576
Fri Mar  9 00:00:41 2012 vfs.zfs.zfetch.block_cap: 256
Fri Mar  9 00:00:41 2012 vfs.zfs.zfetch.min_sec_reap: 2
Fri Mar  9 00:00:41 2012 vfs.zfs.zfetch.max_streams: 8
Fri Mar  9 00:00:41 2012 vfs.zfs.prefetch_disable: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.mg_alloc_failures: 8
Fri Mar  9 00:00:41 2012 vfs.zfs.check_hostid: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.recover: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.txg.synctime_ms: 1000
Fri Mar  9 00:00:41 2012 vfs.zfs.txg.timeout: 5
Fri Mar  9 00:00:41 2012 vfs.zfs.scrub_limit: 10
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.cache.bshift: 16
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.cache.size: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.cache.max: 16384
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.write_gap_limit: 4096
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.read_gap_limit: 32768
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.aggregation_limit: 131072
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.ramp_rate: 2
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.time_shift: 6
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.min_pending: 4
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.max_pending: 10
Fri Mar  9 00:00:41 2012 vfs.zfs.vdev.bio_flush_disable: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.cache_flush_disable: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.zil_replay_disable: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.zio.use_uma: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.version.zpl: 5
Fri Mar  9 00:00:41 2012 vfs.zfs.version.spa: 28
Fri Mar  9 00:00:41 2012 vfs.zfs.version.acl: 1
Fri Mar  9 00:00:41 2012 vfs.zfs.debug: 0
Fri Mar  9 00:00:41 2012 vfs.zfs.super_owner: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hits: 112836151
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.misses: 42194232
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.demand_data_hits: 8223782
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.demand_data_misses: 2595054
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.demand_metadata_hits: 99086549
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.demand_metadata_misses: 19695660
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.prefetch_data_hits: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.prefetch_data_misses: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 5525820
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 19903518
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.mru_hits: 33185334
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.mru_ghost_hits: 992072
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.mfu_hits: 74389339
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.mfu_ghost_hits: 9275805
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.allocated: 71980380
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.deleted: 47758856
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.stolen: 34276801
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.recycle_miss: 14345121
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.mutex_miss: 31907
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.evict_skip: 203612401
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.evict_l2_cached: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.evict_l2_eligible: 549125139456
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.evict_l2_ineligible: 326250878976
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hash_elements: 16239
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hash_elements_max: 166243
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hash_collisions: 31543544
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hash_chains: 2864
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hash_chain_max: 34
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.p: 85759638
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.c: 159243058
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.c_min: 134217728
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.c_max: 1073741824
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.size: 159377760
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.hdr_size: 4036680
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.data_size: 89853440
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.other_size: 65487640
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_hits: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_misses: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_feeds: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_rw_clash: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_read_bytes: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_bytes: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_writes_sent: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_writes_done: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_writes_error: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_evict_reading: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_free_on_write: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_cksum_bad: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_io_error: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_size: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_hdr_size: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.memory_throttle_count: 3411
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_in_l2: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 19944493
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_full: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_pios: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0
Fri Mar  9 00:00:41 2012 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0
```


----------



## belon_cfy (Mar 9, 2012)

phoenix said:
			
		

> Uhm, *32 MB* of swap use is not "heavy swap usage".



Hi Phoenix,
Not "heavy usage", but it keeps swapping in and out continuously until server crashed.

Possible caused by the nfsd because I have increased the number of server to 128 instead of 4. 

```
nfs_server_flags="-u -t -n 128"
```

But stopping nfsd won't reclaim back my memory


----------



## belon_cfy (Mar 13, 2012)

Suspecting it might be due to insufficient memory for holding more than 3TB data with a storage contains 4GB Ram. According to the system requirement of Nexenta, they recommend at least 4-8GB ram for production server and additional 1GB ram is needed for every 1TB space added. 

I have upgraded 1 of the box to 8GB ram and it survives for more than 4 days. Don't know whether will it crashes again in the next few days or weeks.


----------



## DutchDaemon (Mar 13, 2012)

Let us know, so we can mark this [Solved] (or mark it as such yourself).


----------



## belon_cfy (Mar 14, 2012)

DutchDaemon said:
			
		

> Let us know, so we can mark this [Solved] (or mark it as such yourself).



Sure, I will update the status few days later.


----------



## belon_cfy (Mar 15, 2012)

```
CPU:  0.4% user,  0.0% nice,  2.0% system,  0.2% interrupt, 97.5% idle
Mem: 4044K Active, 160K Inact, 7607M Wired, 19M Cache, 182M Free
Swap: 8192M Total, 42M Used, 8150M Free, 1304K Out
```

The server with 8GB RAM keeps swapping out continuously by today. Seems it is going down soon because of the same symptom is happening. 

Moving swap vol from zvol to USB with freebsd-swap didn't solve the problem. My server still down with this swap configuration last time.

I have tried the following configuration but non of these can eliminate servers crashed.

Changing arc_max to below 1GB on 4GB servers.
Avoid swap partition on zvol
Increase kmem_size to 6GB on 4GB server.
Decrease kmem_size to 3GB on 4GB servers.


----------



## peetaur (Mar 15, 2012)

Try setting vm.kmem_size_max to 1 or 2 GB below your physical memory. (or lower if you have memory hogging things in kernel space, like 10Gbps network drivers)

And please repost your zfs-status during the near-melt-down so we can compare numbers (eg. is arc usage higher than arc_max you set?)


----------



## belon_cfy (Mar 15, 2012)

peetaur said:
			
		

> Try setting vm.kmem_size_max to 1 or 2 GB below your physical memory. (or lower if you have memory hogging things in kernel space, like 10Gbps network drivers)
> 
> And please repost your zfs-status during the near-melt-down so we can compare numbers (eg. is arc usage higher than arc_max you set?)



Hi
The server with 2 x 1Gbps network port.

below is my last zfs-stats before it crashed.


```
------------------------------------------------------------------------
 ZFS Subsystem Report                           
 ------------------------------------------------------------------------

 System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

 FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root
 11:39PM  up 3 days, 12:40, 0 users, load averages: 0.53, 0.54, 0.68

 ------------------------------------------------------------------------

 System Memory:

        0.10%   3.79    MiB Active,     0.01%   472.00  KiB Inact
        99.20%  3.81    GiB Wired,      0.15%   5.82    MiB Cache
        0.53%   20.75   MiB Free,       0.02%   808.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 99.49%  3.98    GiB
        Real Managed:                   96.49%  3.84    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   99.34%  3.97    GiB
        Logical Free:                   0.66%   27.03   MiB

 Kernel Memory:                                 396.07  MiB
        Data:                           94.99%  376.23  MiB
        Text:                           5.01%   19.84   MiB

 Kernel Memory Map:                             3.27    GiB
        Size:                           7.93%   265.52  MiB
        Free:                           92.07%  3.01    GiB

 ------------------------------------------------------------------------

 ARC Summary: (THROTTLED)
        Memory Throttle Count:                  4.79k

 ARC Misc:
        Deleted:                                34.77m
        Recycle Misses:                         5.20m
        Mutex Misses:                           241.28k
        Evict Skips:                            241.28k

 ARC Size:                              12.50%  363.44  MiB
        Target Size: (Adaptive)         12.50%  363.48  MiB
        Min Size (Hard Limit):          12.50%  363.48  MiB
        Max Size (High Water):          8:1     2.84    GiB

 ARC Size Breakdown:
        Recently Used Cache Size:       6.35%   23.08   MiB
        Frequently Used Cache Size:     93.65%  340.39  MiB

 ARC Hash Breakdown:
        Elements Max:                           332.55k
        Elements Current:               42.63%  141.77k
        Collisions:                             30.20m
        Chain Max:                              60
        Chains:                                 36.87k

 ------------------------------------------------------------------------

 ARC Efficiency:                                        357.24m
        Cache Hit Ratio:                91.29%  326.11m
        Cache Miss Ratio:               8.71%   31.13m
        Actual Hit Ratio:               91.20%  325.82m

        Data Demand Efficiency:         88.75%  10.72m
        Data Prefetch Efficiency:       0.00%   1.71m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           54.05%  176.28m
          Most Frequently Used:         45.86%  149.54m
          Most Recently Used Ghost:     0.14%   457.14k
          Most Frequently Used Ghost:   1.73%   5.65m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  2.92%   9.52m
          Prefetch Data:                0.00%   31
          Demand Metadata:              96.99%  316.30m
          Prefetch Metadata:            0.09%   289.97k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  3.88%   1.21m
          Prefetch Data:                5.51%   1.71m
          Demand Metadata:              89.59%  27.89m
          Prefetch Metadata:            1.03%   320.32k

 ------------------------------------------------------------------------

 L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        0
        Tried Lock Failures:                    358
        IO In Progress:                         0
        Low Memory Aborts:                      1.29k
        Free on Write:                          10.68k
        Writes While Full:                      136
        R/W Clashes:                            10
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           0

 L2 ARC Size: (Adaptive)                                1.17    GiB
        Header Size:                    1.50%   17.95   MiB

 L2 ARC Breakdown:                              986.25k
        Hit Ratio:                      18.91%  186.55k
        Miss Ratio:                     81.09%  799.70k
        Feeds:                                  200

 L2 ARC Buffer:
        Bytes Scanned:                          7.70    GiB
        Buffer Iterations:                      200
        List Iterations:                        8.84k
        NULL List Iterations:                   2.73k

 L2 ARC Writes:
        Writes Sent:                    100.00% 200

 ------------------------------------------------------------------------


 ------------------------------------------------------------------------

 VDEV cache is disabled

 ------------------------------------------------------------------------

 ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            4122816512
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   1041817088
        vfs.zfs.mfu_ghost_data_lsize            0
        vfs.zfs.mfu_ghost_metadata_lsize        44313088
        vfs.zfs.mfu_ghost_size                  44313088
        vfs.zfs.mfu_data_lsize                  17408
        vfs.zfs.mfu_metadata_lsize              56710656
        vfs.zfs.mfu_size                        204380672
        vfs.zfs.mru_ghost_data_lsize            157184
        vfs.zfs.mru_ghost_metadata_lsize        336623616
        vfs.zfs.mru_ghost_size                  336780800
        vfs.zfs.mru_data_lsize                  0
        vfs.zfs.mru_metadata_lsize              0
        vfs.zfs.mru_size                        43282432
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       1082368
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  762268672
        vfs.zfs.arc_meta_used                   379941128
        vfs.zfs.arc_min                         381134336
        vfs.zfs.arc_max                         3049074688
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            12818632704
        vfs.zfs.write_limit_max                 534109696
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

 ------------------------------------------------------------------------
```


----------



## belon_cfy (Mar 16, 2012)

The storage with 8GB RAM finally down again after 6 days. Adding RAM just make it survives longer but still can't avoid memory exhaustion.


----------



## mamalos (Mar 16, 2012)

Just out of curiosity (bacause I was reading it in another thread): you don't have any tmpfs filesystems mounted, I suppose...


----------



## belon_cfy (Mar 16, 2012)

Seems storing a lot of small file such as email will cause the server consume all of the memory. The small files are rsync from my production servers. 

Another box with vmware images are working fine for more than 25 days. These 2 servers specification are identical. Seems storing different kind of data will result to different stability on FreeBSD + ZFS.


----------



## peetaur (Mar 16, 2012)

```
Kernel Memory:                                 396.07  MiB
 ARC Size:                              12.50%  363.44  MiB
        vm.kmem_size                            4122816512
```

How terribly strange. Why should zfs-status show a different number. 

Please show output from:

`# grep -vE "^#|^$" /boot/loader.conf`
`# grep -vE "^#|^$" /etc/sysctl.conf`

And actually, if you trust the numbers from zfs-status, I would say ZFS may be crashing in the end, due to zero memory, but doesn't actually seem to be the cause of the leak. I'm not sure what the right command to check is... but check the man pages for netstat to find out how to print memory stats, and post results here.

Probably:
`# netstat -m`

Also systat might have something more interesting. (netstat would only tell you if it was your network using the memory)

Also check top or use procstat to find which user space process uses the most memory. But I think we are looking for something eating kernel memory... something different. I don't know if top and procstat tell you anything about that.

I'm not a low level guy, like a FreeBSD developer, so I don't know where to point you to find the root of the problem. Can someone else provide suggestions?


----------



## belon_cfy (Mar 18, 2012)

Hi Peetaur

Below is my *netstat -m* nearly three days uptime. Seems nothing wrong with the memory usage.

```
263/3667/3930 mbufs in use (current/cache/total)
260/2162/2422/25600 mbuf clusters in use (current/cache/total/max)
260/932 mbuf+clusters out of packet secondary zone in use (current/cache)
0/44/44/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
585K/5416K/6002K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
```

I have disabled primarycache on all zfs volumes. But it still consume almost all of my memory.

```
Mem: 8804K Active, 26M Inact, 3448M Wired, 544K Cache, 353M Free
Swap: 8192M Total, 1192K Used, 8191M Free
```

procstat -a showing the following result:

```
PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
    0     0     0     0     0 166 -        -         -             kernel
    1     0     1     1     0   1 root     wait      FreeBSD ELF64 init
    2     0     0     0     0   5 -        zvol:io   -             zfskern
    3     0     0     0     0   1 -        waiting_  -             sctp_iterator
    4     0     0     0     0   1 -        ccb_scan  -             xpt_thrd
    5     0     0     0     0   1 -        psleep    -             pagedaemon
    6     0     0     0     0   1 -        psleep    -             vmdaemon
    7     0     0     0     0   1 -        pgzero    -             pagezero
    8     0     0     0     0   1 -        psleep    -             bufdaemon
    9     0     0     0     0   1 -        syncer    -             syncer
   10     0     0     0     0   1 -        audit_wo  -             audit
   11     0     0     0     0   4 -        -         -             idle
   12     0     0     0     0  19 -        -         -             intr
   13     0     0     0     0   3 -        -         -             geom
   14     0     0     0     0   1 -        -         -             yarrow
   15     0     0     0     0   8 -        -         -             usb
   16     0     0     0     0   1 -        vlruwt    -             vnlru
   17     0     0     0     0   1 -        sdflush   -             softdepflush
  840     1   840   840     0   1 root     select    FreeBSD ELF64 devd
 1015     1  1015  1015     0   1 root     select    FreeBSD ELF64 syslogd
 1038     1  1038  1038     0   1 root     select    FreeBSD ELF64 rpcbind
 1162     1  1162  1162     0   1 root     pause     FreeBSD ELF64 nfsuserd
 1163  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1164  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1165  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1166  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1178     1  1178  1178     0   1 root     select    FreeBSD ELF64 mountd
 1180     1  1180  1180     0   1 root     select    FreeBSD ELF64 nfsd
 1181  1180  1180  1180     0 128 root     rpcsvc    FreeBSD ELF64 nfsd
 1187     1  1187  1187     0   1 root     select    FreeBSD ELF64 rpc.statd
 1193     1  1193  1193     0   1 root     rpcsvc    FreeBSD ELF64 rpc.lockd
 1320     1  1320  1320     0   1 root     select    FreeBSD ELF64 sshd
 1327     1  1327  1327     0   1 root     select    FreeBSD ELF64 sendmail
 1333     1  1333  1333     0   1 root     pause     FreeBSD ELF64 sendmail
 1339     1  1339  1339     0   1 root     nanslp    FreeBSD ELF64 cron
 1405     1  1405  1405  1405   1 root     ttyin     FreeBSD ELF64 getty
 1406     1  1406  1406  1406   1 root     ttyin     FreeBSD ELF64 getty
 1407     1  1407  1407  1407   1 root     ttyin     FreeBSD ELF64 getty
 1408     1  1408  1408  1408   1 root     ttyin     FreeBSD ELF64 getty
 1409     1  1409  1409  1409   1 root     ttyin     FreeBSD ELF64 getty
 1410     1  1410  1410  1410   1 root     ttyin     FreeBSD ELF64 getty
 1411     1  1411  1411  1411   1 root     ttyin     FreeBSD ELF64 getty
 1412     1  1412  1412  1412   1 root     ttyin     FreeBSD ELF64 getty
 1573  1320  1573  1573     0   1 root     select    FreeBSD ELF64 sshd
 1576  1573  1576  1576  1576   1 root     pause     FreeBSD ELF64 csh
 1707  1320  1707  1707     0   1 root     select    FreeBSD ELF64 sshd
 1710  1707  1710  1710  1710   1 root     pause     FreeBSD ELF64 csh
 1714  1710  1714  1710  1710   1 root     select    FreeBSD ELF64 top
 1715  1320  1715  1715     0   1 root     select    FreeBSD ELF64 sshd
 1742  1715  1742  1742  1742   1 root     pause     FreeBSD ELF64 csh
 1747  1742  1747  1742  1742   1 root     nanslp    FreeBSD ELF64 vmstat
 1748  1320  1748  1748     0   1 root     select    FreeBSD ELF64 sshd
 1751  1748  1751  1751  1751   1 root     pause     FreeBSD ELF64 csh
 1755  1751  1755  1751  1751   1 root     nanslp    FreeBSD ELF64 perl5.12.4
 1778  1320  1778  1778     0   1 root     select    FreeBSD ELF64 sshd
 1810  1778  1810  1810  1810   1 root     pause     FreeBSD ELF64 csh
 1830  1810  1830  1810  1810   1 root     nanslp    FreeBSD ELF64 zpool
11070  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11071  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11072  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11073  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11074 11071 11074 11074     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11075 11070 11075 11075     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11076 11072 11076 11076     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11077 11073 11077 11077     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11142  1576 11142  1576  1576   1 root     -         FreeBSD ELF64 procstat
32598  1320 32598 32598     0   1 root     select    FreeBSD ELF64 sshd
32613 32598 32613 32613 32613   1 root     ttyin     FreeBSD ELF64 csh
```


----------



## belon_cfy (Mar 18, 2012)

Continue the post above
/etc/rc.conf

```
dumpdev="AUTO"
sshd_enable="YES"

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_flags="-u -t -n 128"
```

/boot/loader.conf

```
zfs_enable="YES"
storage20# more /boot/loader.conf
if_vlan_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
```


*zfs-stats -a* result as below:

```
------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Mar 18 18:10:25 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root
 6:10PM  up 2 days,  4:59, 6 users, load averages: 0.01, 0.02, 0.00

------------------------------------------------------------------------

System Memory:

        0.25%   9.50    MiB Active,     0.67%   25.70   MiB Inact
        89.89%  3.37    GiB Wired,      0.01%   544.00  KiB Cache
        9.16%   351.48  MiB Free,       0.02%   860.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 97.36%  3.89    GiB
        Real Managed:                   96.22%  3.75    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   90.78%  3.63    GiB
        Logical Free:                   9.22%   377.71  MiB

Kernel Memory:                                  1.05    GiB
        Data:                           98.16%  1.03    GiB
        Text:                           1.84%   19.84   MiB

Kernel Memory Map:                              3.20    GiB
        Size:                           30.14%  987.71  MiB
        Free:                           69.86%  2.24    GiB

------------------------------------------------------------------------

ARC Summary: (THROTTLED)
        Memory Throttle Count:                  8

ARC Misc:
        Deleted:                                13.99m
        Recycle Misses:                         1.40k
        Mutex Misses:                           70.55k
        Evict Skips:                            70.55k

ARC Size:                               37.99%  1.04    GiB
        Target Size: (Adaptive)         45.24%  1.24    GiB
        Min Size (Hard Limit):          12.50%  351.65  MiB
        Max Size (High Water):          8:1     2.75    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       91.56%  1.14    GiB
        Frequently Used Cache Size:     8.44%   107.36  MiB

ARC Hash Breakdown:
        Elements Max:                           377.32k
        Elements Current:               40.02%  150.99k
        Collisions:                             13.48m
        Chain Max:                              62
        Chains:                                 21.92k

------------------------------------------------------------------------

ARC Efficiency:                                 99.58m
        Cache Hit Ratio:                51.01%  50.80m
        Cache Miss Ratio:               48.99%  48.79m
        Actual Hit Ratio:               50.91%  50.70m

        Data Demand Efficiency:         0.26%   6.40m
        Data Prefetch Efficiency:       0.00%   175

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           1.18%   601.49k
          Most Frequently Used:         98.62%  50.10m
          Most Recently Used Ghost:     7.18%   3.65m
          Most Frequently Used Ghost:   75.07%  38.14m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  0.03%   16.43k
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.25%  50.42m
          Prefetch Metadata:            0.72%   365.62k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  13.08%  6.38m
          Prefetch Data:                0.00%   175
          Demand Metadata:              86.92%  42.41m
          Prefetch Metadata:            0.00%   168

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------


------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            4023611392
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            4263424
        vfs.zfs.mfu_ghost_metadata_lsize        913382912
        vfs.zfs.mfu_ghost_size                  917646336
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              275456
        vfs.zfs.mfu_size                        1801216
        vfs.zfs.mru_ghost_data_lsize            123222016
        vfs.zfs.mru_ghost_metadata_lsize        182882304
        vfs.zfs.mru_ghost_size                  306104320
        vfs.zfs.mru_data_lsize                  514520064
        vfs.zfs.mru_metadata_lsize              479475712
        vfs.zfs.mru_size                        1027247104
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       409600
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  737467392
        vfs.zfs.arc_meta_used                   605950848
        vfs.zfs.arc_min                         368733696
        vfs.zfs.arc_max                         2949869568
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            12544475136
        vfs.zfs.write_limit_max                 522686464
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------
```

My *zpool list* as below, already 83% of space is in used. 

```
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  3.62T  3.01T   628G    83%  1.00x  ONLINE  -
```

This is my backup server storing a lot of emails and web contents with daily rsync. What's wrong with the FreeBSD + ZFS on storing such data in this scenario? I have two servers with completely different hardware but storing similar data up to 3TB per server are facing the exactly same problem. This is the only reason stopping me to use FreeBSD + ZFS in production environment.


----------



## peetaur (Mar 18, 2012)

Have you tried using 8-STABLE or 8.3-RC1?


----------



## belon_cfy (Mar 19, 2012)

Hi Peetaur
There is no way for me to downgrade to FreeBSD 8.3. I don't have any free server to reinstall for migration.

Below is my latest memory usage, I still can't find any process consume all of my memory.


```
last pid: 24556;  load averages:  0.17,  0.10,  0.03  up 3+10:19:14    10:26:12
60 processes:  1 running, 59 sleeping

Mem: 7940K Active, 7288K Inact, 3664M Wired, 1252K Cache, 251M Free
Swap: 8192M Total, 19M Used, 8173M Free


  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    CPU COMMAND
66237 root      20    0 18404K   844K nanslp  1  63:36  0.39% vmstat 0.5
 1703 root      20    0 10052K   812K rpcsvc  0   6:30  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:30  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  3   6:27  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:26  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:25  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:22  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:22  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:21  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:19  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:18  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:17  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:17  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:15  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:13  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  3   6:12  0.00% nfsd: server (nfsd){nfsd: master}
 1703 root      20    0 10052K   812K rpcsvc  1   6:11  0.00% nfsd: server (nfsd){nfsd: service}
66245 root      52    0 19964K  1360K nanslp  0   4:37  0.00% /usr/bin/perl /root/zfs_list.pl (perl5.12.4)
66307 root      20    0 35604K  1280K nanslp  1   3:45  0.00% zpool iostat -v 1
66238 root      20    0 68016K  1252K select  3   1:34  0.00% sshd: root@pts/3 (sshd)
66206 root      20    0 68016K  1244K select  0   0:44  0.00% sshd: root@pts/2 (sshd)
66255 root      20    0 68016K  1268K select  2   0:38  0.00% sshd: root@pts/4 (sshd)
 1849 root      20    0 20384K  1168K select  0   0:04  0.00% sendmail: accepting connections (sendmail)
66157 root      20    0 68016K  1164K select  0   0:04  0.00% sshd: root@pts/0 (sshd)
 1861 root      20    0 14260K   452K nanslp  0   0:04  0.00% /usr/sbin/cron -s
 1537 root      20    0 12184K   792K select  3   0:02  0.00% /usr/sbin/syslogd -s
 1560 root      20    0 14264K   760K select  0   0:01  0.00% /usr/sbin/rpcbind
 1715 root      52    0 14264K   612K rpcsvc  0   0:00  0.00% /usr/sbin/rpc.lockd
 1709 root      20    0   268M   684K select  3   0:00  0.00% /usr/sbin/rpc.statd
 1686 root      20    0 10052K   660K select  2   0:00  0.00% nfsuserd: slave (nfsuserd)
 1687 root      20    0 10052K   660K select  0   0:00  0.00% nfsuserd: slave (nfsuserd)
66160 root      20    0 14612K  1524K pause   0   0:00  0.00% -csh (csh)
 1685 root      20    0 10052K   660K select  1   0:00  0.00% nfsuserd: slave (nfsuserd)
 1688 root      20    0 10052K   660K select  1   0:00  0.00% nfsuserd: slave (nfsuserd)
 1855 smmsp     20    0 20384K   424K pause   2   0:00  0.00% sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (
 1700 root      20    0 12180K   624K select  1   0:00  0.00% /usr/sbin/mountd -r /etc/exports /etc/zfs/exports
 1702 root      20    0 10052K   580K select  0   0:00  0.00% nfsd: master (nfsd)
66241 root      22    0 14612K     0K pause   1   0:00  0.00% -csh (<csh>)
66270 root      22    0 14612K     0K pause   0   0:00  0.00% -csh (<csh>)
66233 root      22    0 14612K     0K pause   0   0:00  0.00% -csh (<csh>)
24493 root      36    0 19964K  2348K nanslp  0   0:00  0.00% /usr/bin/perl /root/perl_zfs_sysctl.pl (perl5.12.4)
24495 root      22    0 19964K  2320K nanslp  3   0:00  0.00% /usr/bin/perl /root/perl_uptime.pl (perl5.12.4)
24492 root      52    0 19964K  2356K nanslp  0   0:00  0.00% /usr/bin/perl /root/perl_zfs_stats.pl (perl5.12.4)
24494 root      27    0 19964K  2320K nanslp  3   0:00  0.00% /usr/bin/perl /root/perl_memstatus.pl (perl5.12.4)
 1842 root      20    0 46876K   880K select  3   0:00  0.00% /usr/sbin/sshd
 1342 root      20    0 10372K   340K select  0   0:00  0.00% /sbin/devd
 1923 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv4
24556 root      20    0 16700K  1888K CPU1    1   0:00  0.00% top -naCH 100000
 1919 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv0
 1920 root      52    0 12184K   580K ttyin   2   0:00  0.00% /usr/libexec/getty Pc ttyv1
 1925 root      52    0 12184K   580K ttyin   1   0:00  0.00% /usr/libexec/getty Pc ttyv6
 1924 root      52    0 12184K   580K ttyin   3   0:00  0.00% /usr/libexec/getty Pc ttyv5
 1921 root      52    0 12184K   580K ttyin   2   0:00  0.00% /usr/libexec/getty Pc ttyv2
 1922 root      52    0 12184K   580K ttyin   3   0:00  0.00% /usr/libexec/getty Pc ttyv3
 1926 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv7
24489 root      20    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)
 1684 root      34    0 10052K     0K pause   0   0:00  0.00% nfsuserd: master (<nfsuserd>)
24490 root      20    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)
24488 root      20    0 14260K  1076K piperd  0   0:00  0.00% cron: running job (cron)
24491 root      21    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)
```

`netstat -m`

```
2055/2520/4575 mbufs in use (current/cache/total)
2050/1712/3762/25600 mbuf clusters in use (current/cache/total/max)
2050/766 mbuf+clusters out of packet secondary zone in use (current/cache)
0/71/71/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
4613K/4338K/8951K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
```


----------



## peetaur (Mar 19, 2012)

You should be able to just install 8.3 in a new dataset, and then change the bootfs property to point to the right version. You should find a howto somewhere on how to run multiple versions at the same time. And try it in a VM first.


----------



## belon_cfy (Mar 20, 2012)

Changing primarycache to none didn't solve my problem too, it crashed again last night because of memory exhaustion. 

Limiting arc_max, kmem_size, kmem_size_max, swap partition type changed to freebsd-swap, adding 8GB thumbdrive for l2arc, upgrading ram to 8GB, lastly disabled primarycache. All of the methods above didn't solve my problem. 

Any suggestion? Downgrading to FreeBSD 8.3?


----------



## alsuki (Mar 26, 2012)

Hi, everyone.

Just now I saw this thread. I had the same problem about a year back, at that time I didn't know the cause, so I reinstall Linux. At that time I had a AMD64 athlon x2 with 8GB of memory and 5 TB of disk, and every 7 days the machine would hang during the night.

My point is downgrading will not solve the problem. But I would be very glad to see this problem solved, because I'm creating a script for a full install of FreeBS*D* 9.0 on my machine.

Sorry I can*'*t be of any help, but keep up the good work.

Best regards,
Alsuki.


----------



## peetaur (Mar 26, 2012)

alsuki said:
			
		

> Hi, everyone.
> 
> Just now I saw this thread. I had the same problem about a year back, at that time I didn't know the cause, so I reinstall Linux. At that time I had a AMD64 athlon x2 with 8GB of memory and 5 TB of disk, and every 7 days the machine would hang during the night.
> 
> ...



What version did you use? Lots of mysterious problems happen with ZFS in 8.2-RELEASE and earlier, and 8-STABLE until around October 2011. And I wouldn't try 9 in production until it is proven stable.

And some problems are confused for other problems. For example, if you have many snapshots and run a command that goes through all the .zfs/snapshot/* directories, for example: `# find /tank/.zfs/snapshot -type d`, it is well known that your system will fail due to running out of memory. And yet you can run a perfectly stable system if you just simply avoid doing that.

And to the OP, I forget if the above type of problem was ever mentioned, but you could try to figure out if something like that is happening. (not sure how... maybe just make a cronjob that logs "*ps axl*" and "*ps aux*" output, assuming that the problem process will run long enough to find). 

Other random ideas that came to mind:

Maybe exporting and importing the pool or some other similar hack would clean up the memory.

Deleting all but a few snapshots would prevent the above type of problem (I would not be happy with this solution)

You should post a detailed question on the freebsd-current or freebsd-fs mailing lists to get attention of developers, because they would know how to track down the root cause of the problem. You could also try updating to the latest 9 code, in case that is what the freebsd-current people expect you to be running. (but expect some people to be annoyed if you update between questions, because then their diagnostic commands they want you to run won't apply to your new version anymore and give inconsistent information).


----------



## mamalos (Mar 27, 2012)

belon_cfy,

Follow these instructions about debugging a FreeBSD kernel crash, and send the stack trace of *gdb* to the freebsd-stable mailing list. Developers are watching this list too, so it would be very helpful for them to obtain a stack trace of your crashing system. On the other hand, since it's a memory leak, I am not sure if your output would suffice (the output will show which part of the system crashed due to memory starvation -which is a probably a bug- but will not show which function caused the starvation). Anyway, it would be a good idea to talk to these guys, since the lists are much more technical than this forum.


----------



## obrandmueller (Apr 2, 2012)

*FreeBSD 9 amd64, ZFS, NFS*

I'm suspecting a NAMEI leak in most probably NFS or a combination of NFS and ZFS.

I'm seeing constant growth in *vmstat -z* for NAMEI while in other systems (running ZFS, no NFS or running old NFS, no ZFS, FreeBSD 8) I see NAMEI always close to 0 in the USED row.

Could you confirm this for your systems showing the wrong behaviour?

- Oliver


----------



## VVB16 (Apr 3, 2012)

Hi

I ran into similar (or same) problems with FreeBSD 8 from the beginning.
Solution that works for me:

apply patch to kernel:

```
=========================================================================
--- sys/kern/kern_malloc.c      2009-08-03 15:13:06.000000000 +0700
+++ sys/kern/kern_malloc.c      2010-05-19 11:07:09.717752297 +0700
@@ -611,8 +611,10 @@
         * to something sane. Be careful to not overflow the 32bit
         * ints while doing the check.
         */
+/*
        if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count)
                vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE;
+*/

        /*
         * Tune settings based on the kmem map's size at this time.
=========================================================================
```

Add to /boot/loader.conf:

```
vm.kmem_size="16G"
vfs.zfs.arc_min="1G"
vfs.zfs.arc_max="4G"
```

These values used on 8GB RAM server. Main tunable is vm.kmem_size. As far as I can remember real problem is memory fragmentation which leads to memory exhaustion. Patch needed to set such high vm.kmem_size, check it after reboot with *sysctl -a vm.kmem_size*.


----------



## peetaur (Apr 3, 2012)

VVB16, so is the idea to just allocate it all now and not change it, thereby making vm.kmem_size_max obsolete?


----------



## VVB16 (Apr 3, 2012)

peetaur,

No. Actual output from this box:

`sysctl -a |grep vm.kmem_size`


```
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 17179869184
```

Problem with vm.kmem_size was (and probably is) in that it*'*s not allocated at all when needed. Maybe because system has free (virtual) memory, but this memory fragmented and at some time call for memory fails => kernel panic.

At the beginning I set vm.kmem_size to 24G (3*RAM size), later reduced it to 2*RAM just to see how it works. System works stable _with our workload_ with this settings, so I stop tuning it.


----------



## peetaur (Apr 4, 2012)

BTW have you guys seen the "9-STABLE, ZFS, NFS, ggatec - suspected memory leak" thread in the freebsd-stable mailing list?

It sounds like a similar issue to yours. One guy says it is definitely ZFS. Other guy says it is definitely the new NFS server (default in 9.0 I think) and has no problems using the old NFS server (default in 8.2).

And the memory usage can be seen using `# vmstat -z`


----------



## belon_cfy (May 14, 2012)

I'm trying the following patch from now, will report 2-3 days later whether the problem still persists.

http://people.freebsd.org/~rmacklem/namei-leak.patch

```
--- fs/nfsserver/nfs_nfsdport.c.sav	2012-04-25 16:50:05.000000000 -0400
+++ fs/nfsserver/nfs_nfsdport.c	2012-04-25 17:08:43.000000000 -0400
@@ -1047,6 +1047,8 @@ nfsvno_removesub(struct nameidata *ndp, 
 	else
 		vput(ndp->ni_dvp);
 	vput(vp);
+	if ((ndp->ni_cnd.cn_flags & SAVENAME) != 0)
+		nfsvno_relpathbuf(ndp);
 	NFSEXITCODE(error);
 	return (error);
 }
@@ -1086,6 +1088,8 @@ out:
 	else
 		vput(ndp->ni_dvp);
 	vput(vp);
+	if ((ndp->ni_cnd.cn_flags & SAVENAME) != 0)
+		nfsvno_relpathbuf(ndp);
 	NFSEXITCODE(error);
 	return (error);
 }
```


----------



## belon_cfy (Jun 21, 2012)

Seems the problem has been resolved by applying the patch above. My servers are running for more than two days, however the memory is still able to maintain 2GB free after flushing it. 

I can assume that the memory leak issue is gone. 

Below are the steps of applying the patch:

```
pkg_add -r wget
cd /usr/src/sys
/usr/local/bin/wget http://people.freebsd.org/~rmacklem/namei-leak.patch
patch < namei-leak.patch
cd /usr/src
make buildkernel
make installkernel

reboot
```

http://people.freebsd.org/~rmacklem/namei-leak.patch

```
--- fs/nfsserver/nfs_nfsdport.c.sav	2012-04-25 16:50:05.000000000 -0400
+++ fs/nfsserver/nfs_nfsdport.c	2012-04-25 17:08:43.000000000 -0400
@@ -1047,6 +1047,8 @@ nfsvno_removesub(struct nameidata *ndp, 
 	else
 		vput(ndp->ni_dvp);
 	vput(vp);
+	if ((ndp->ni_cnd.cn_flags & SAVENAME) != 0)
+		nfsvno_relpathbuf(ndp);
 	NFSEXITCODE(error);
 	return (error);
 }
@@ -1086,6 +1088,8 @@ out:
 	else
 		vput(ndp->ni_dvp);
 	vput(vp);
+	if ((ndp->ni_cnd.cn_flags & SAVENAME) != 0)
+		nfsvno_relpathbuf(ndp);
 	NFSEXITCODE(error);
 	return (error);
 }
```

May I know: will it merge into the next release of FreeBSD 9?


----------



## kpa (Jun 21, 2012)

It's in 9-STABLE so it will make it into 9.1-RELEASE.


----------

