# Inactive memory not reallocated - swapping



## rev (Mar 27, 2014)

Hello,

My system does not reallocate inactive memory and is swapping. `swapoff -a` fails with allocation error (as expected). Inactive memory seems to get to the current level after a few days and stays there.

Machine has 32G RAM. Output of `top -n`:

```
113 processes: 1 running, 112 sleeping

Mem: 251M Active, 27G Inact, 3593M Wired, 513M Cache, 1660M Buf, 165M Free
Swap: 4096M Total, 712M Used, 3384M Free, 17% Inuse


  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1459 vbox           24  21    0  1394M  1106M select  1 431:38   4.30% VBoxHeadless
  809 transmission    3  20    0   250M 42028K select  1  28:31   0.10% transmission-daemon
 1294 mysql          25  20    0   802M   189M uwait   0   7:56   0.00% mysqld
22738 www             1  24    0   241M     0K accept  2   0:52   0.00% <php-fpm>
24649 www             1  24    0   229M     0K accept  4   0:43   0.00% <php-fpm>
  805 vbox           11  20    0   140M  4976K uwait   0   0:31   0.00% VBoxSVC
24636 www             1  24    0   221M     0K accept  5   0:17   0.00% <php-fpm>
  803 vbox            1  20    0 83324K  3596K select  0   0:09   0.00% VBoxXPCOMIPCD
 1015 pgsql           1  20    0  4284M  3724K select  5   0:06   0.00% postgres
 1021 pgsql           1  20    0 70432K  3960K select  7   0:06   0.00% postgres
 1017 pgsql           1  20    0  4284M 31936K select  1   0:04   0.00% postgres
  763 root            1  20    0 25328K  2136K select  7   0:04   0.00% ntpd
 1020 pgsql           1  20    0  4284M  4144K select  1   0:04   0.00% postgres
  819 root            2  20    0 18964K   604K nanslp  1   0:04   0.00% sshguard
 1315 jabber          1  20    0 47036K  5644K select  3   0:03   0.00% s2s
97487 www             1  20    0 32592K  2824K kqread  5   0:02   0.00% nginx
  536 _pflogd         1  20    0 14600K  1028K bpf     2   0:02   0.00% pflogd
  788 vbox           11  20    0   157M  4728K uwait   7   0:02   0.00% vboxwebsrv
```
`uname -a`:

```
FreeBSD x.y.z 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014     root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```
Daemons running:

 postgresql
 mysql
 transmission
 dovecot
 postfix
 spamassassin
 virtualbox-headless + web
 gopherd
 jabberd2
 nginx
 php-fpm
 sshguard
Output from a freebsd-memory perl script from Ralf S. Engelschall:

```
SYSTEM MEMORY INFORMATION:
mem_wire:        3772944384 (   3598MB) [ 11%] Wired: disabled for paging out
mem_active:  +    267919360 (    255MB) [  0%] Active: recently referenced
mem_inactive:+  28555124736 (  27232MB) [ 85%] Inactive: recently not referenced
mem_cache:   +    533000192 (    508MB) [  1%] Cached: almost avail. for allocation
mem_free:    +    173142016 (    165MB) [  0%] Free: fully available for allocation
mem_gap_vm:  +      1081344 (      1MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =  33303212032 (  31760MB) [100%] Total real memory managed
mem_gap_sys: +    888053760 (    846MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =  34191265792 (  32607MB)        Total real memory available
mem_gap_hw:  +    168472576 (    160MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =  34359738368 (  32768MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        5098471424 (   4862MB) [ 14%] Logically used memory
mem_avail:   +  29261266944 (  27905MB) [ 85%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =  34359738368 (  32768MB) [100%] Logically total memory
```

I have no idea what can cause this kind of behaviour. Can anyone help?


----------



## phoenix (Mar 28, 2014)

Are you running ZFS?  There's a recent kernel PR that details issues with the ARC, wired memory, inactive memory, and swapping.


----------



## rev (Mar 28, 2014)

No, using UFS.


----------



## phoenix (Mar 28, 2014)

When in top, press 'o' to change the column to order on.  Try "res" for resident memory usage, and "size" for memory size.  See if that helps bring the memory hog to the top of the list.  That should help track down the process using all the RAM.  Then it's a matter of figuring out why it's holding onto it all.

I have a feeling it will be Postgres (4 GB per process showing in your original top output, although that could be shared).


----------



## rev (Mar 28, 2014)

phoenix said:
			
		

> When in top, press 'o' to change the column to order on.  Try "res" for resident memory usage, and "size" for memory size.  See if that helps bring the memory hog to the top of the list.  That should help track down the process using all the RAM.  Then it's a matter of figuring out why it's holding onto it all.


There is no process using that much memory.
`top -n -o res`

```
PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1459 vbox           24  21    0  1406M  1118M select  1 547:07   4.39% VBoxHeadless
 1294 mysql          25  21    0   814M   168M uwait   6   9:58   0.00% mysqld
  809 transmission    3  20    0   258M 38420K select  2  36:53   0.10% transmission-daemon
 1017 pgsql           1  20    0  4284M 32488K select  6   0:06   0.00% postgres
93068 root            1  20    0   204M 11416K kqread  2   0:01   0.00% php-fpm
48665 root            1  20    0 86084K  5164K select  3   0:00   0.00% sshd
48667 phoenix         1  20    0 86084K  5160K select  4   0:00   0.00% sshd
  805 vbox           11  20    0   140M  5100K uwait   5   0:39   0.00% VBoxSVC
  788 vbox           11  20    0   157M  4676K uwait   7   0:03   0.00% vboxwebsrv
 1315 jabber          1  20    0 47036K  4524K select  5   0:04   0.00% s2s
 1462 vbox            1  20    0 84416K  4272K VBoxIS  6   0:01   0.00% VBoxNetDHCP
 1314 jabber          1  20    0 47256K  4168K select  7   0:01   0.00% c2s
 1020 pgsql           1  20    0  4284M  4080K select  0   0:05   0.00% postgres
 1021 pgsql           1  20    0 70432K  3960K select  6   0:08   0.00% postgres
 1015 pgsql           1  20    0  4284M  3704K select  0   0:08   0.00% postgres
 1018 pgsql           1  20    0  4284M  3584K select  3   0:01   0.00% postgres
  803 vbox            1  20    0 83324K  3584K select  3   0:11   0.00% VBoxXPCOMIPCD
 1019 pgsql           1  20    0  4284M  3568K select  6   0:01   0.00% postgres
```

`top -n -o size`

```
PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1353 pgsql           1  20    0  4288M     0K sbwait  3   0:00   0.00% <postgres>
 1352 pgsql           1  20    0  4288M     0K sbwait  2   0:00   0.00% <postgres>
48746 pgsql           1  20    0  4288M     0K sbwait  7   0:00   0.00% <postgres>
 1017 pgsql           1  20    0  4284M 32488K select  6   0:06   0.00% postgres
 1020 pgsql           1  20    0  4284M  4080K select  3   0:05   0.00% postgres
 1015 pgsql           1  20    0  4284M  3704K select  4   0:08   0.00% postgres
 1018 pgsql           1  20    0  4284M  3584K select  0   0:01   0.00% postgres
 1019 pgsql           1  20    0  4284M  3568K select  4   0:01   0.00% postgres
 1459 vbox           24  21    0  1406M  1118M select  1 547:09   4.05% VBoxHeadless
 1294 mysql          25  20    0   814M   168M uwait   4   9:58   0.00% mysqld
  809 transmission    3  20    0   258M 38372K select  3  36:53   0.10% transmission-daemon
47163 www             1  23    0   237M 11760K accept  7   0:16   0.00% php-fpm
47137 www             1  20    0   233M 10188K accept  7   0:08   0.00% php-fpm
47277 www             1  52    0   229M     0K accept  4   0:10   0.00% <php-fpm>
93068 root            1  20    0   204M 11416K kqread  5   0:01   0.00% php-fpm
  788 vbox           11  20    0   157M  4676K uwait   7   0:03   0.00% vboxwebsrv
  805 vbox           11  20    0   140M  5100K uwait   5   0:39   0.00% VBoxSVC
 1313 jabber          1  20    0 96728K  3364K select  0   0:02   0.00% sm
```



			
				phoenix said:
			
		

> I have a feeling it will be Postgres (4 GB per process showing in your original top output, although that could be shared).


Yes, it's shared memory and not allocated per process.


----------



## Crivens (Mar 29, 2014)

Could that be from transmission? Things like network buffers and that?

For a brute force debugging I would suggest to restart all services, one after the other, and check if the memory comes back. But that may be something the users may not want to happen, but then - who wants a system that is turning into a snail day by day?


----------



## rev (Mar 29, 2014)

But how can I see if the memory is back? If I shut a daemon down, I of course get more free memory and can run `swapoff -a` successfully. But that doesn't mean it's using the inactive one.

Or should "inactive" one become "free" again?


----------

