# Visible Memory Shrinking



## gcomeau (Aug 3, 2020)

I have a system with 8 GB of RAM on an AWS instance.


```
root@backslave-main-pr:/usr/home/m2msadmin # uname -a && dmesg | grep memory
FreeBSD backslave-main-pr 12.1-RELEASE-p2 FreeBSD 12.1-RELEASE-p2 GENERIC  amd64
real memory  = 9556721664 (9114 MB)
avail memory = 8170881024 (7792 MB)
```

running MariaDB. Over time, a growing portion of the total memory seems to disappear from FreeBSD's line of sight.

Starting here, `top` reports 8591 MB total

```
root@backslave-main-pr:/usr/home/m2msadmin # top | grep Mem
Mem: 642M Active, 379M Inact, 1230M Wired, 784M Buf, 5556M Free
```

to this, after a few hours.  `top` reports 2284 MB total

```
root@backslave-main-pr:/usr/home/m2msadmin # top  | grep Mem
Mem: 110M Active, 148K Inact, 388K Laundry, 1268M Wired, 783M Buf, 123M Free
```

Where could tbe missing 6GB be?


----------



## Mjölnir (Aug 3, 2020)

Your VM's resources have been reduced because it does not need that much.  Once your VM uses more, it will get migrated to another host and it's resources adjusted.  The automagic of professinal VM server plants works well behind the scenes...


----------



## gcomeau (Aug 3, 2020)

In theory, that memory is available to me if I need it then?

Here's more context on how it happens. Based on what you just said, the little spike in the middle would be the app requesting more RAM and the VM making it available to the OS, until it goes unused again?

I was surprised to see RAM go down while my entire budget is active. Can I infer that memory that FreeBSD sees as inactive gets reclaimed?


----------



## Mjölnir (Aug 3, 2020)

Usually, the VM's OS is aware of the fact that it runs virtualized, and knows how to interact with the host through the use of specialized drivers (virtual memory, disk, etc.).  It is common practice to overcommit on resources, like e.g. an aviation company commonly sells a few more tickets than there are seats in the plane.  As long your applications run well, there's no need to worry.  So the answer should be: yes.


----------



## gcomeau (Aug 6, 2020)

So - I think I may a counterexample to the above, but I don't know how I can reproduce in a small contained example. It could also be specific to my setup somehow.

Here is a memory graph from 3 database server (mariadb 10.4 on FreeBSD 12.1-RELEASE-p2 GENERIC  amd64)

See how the active memory on both slaves gradually shrinks in each case, up to the point of crashes.

`Aug  5 00:37:45 backslave-main-pr kernel: pid 10496 (mysqld), jid 0, uid 88, was killed: out of swap space
Aug  6 05:45:52 backslave-main-pr kernel: pid 55545 (mysqld), jid 0, uid 88, was killed: out of swap space
Aug  6 05:45:56 backslave-main-pr kernel: pid 77367 (perl), jid 0, uid 65534, was killed: out of swap space
Aug  6 05:45:57 backslave-main-pr kernel: pid 686 (ntpd), jid 0, uid 123, was killed: out of swap space`

Memory never goes inactive - always from active to missing, until the point where the OS can't perform an allocation. 

Note that since these are all fruits of the same tree, I realize the DB & schema are suspect but since the OS is showing an ever shrinking memory space alotted to the DB, I am not sure where to go next. What can I do to further isolate the issue? Would a memory leak manifest itself as inactive memory or 'VM reclaimed'  memory.

I'll think I can grow one of the slaves to quadruple memory and see how the behavior changes.


----------



## Mjölnir (Aug 6, 2020)

Insistently ask your VM provider's staff about this.  If they start to complain: your system crashes, that should not happen.  You've got a contract including an assertion on available resources (espc. RAM memory).


----------



## gcomeau (Aug 6, 2020)

Following up on AWS Forums.


----------



## Mjölnir (Aug 6, 2020)

You should also `freebsd-update` your systems, we're now at -p8.


----------



## gcomeau (Aug 8, 2020)

I will roll the -p8. 

Now what to make of this: 


`last pid:  6910;  load averages:  3.70,  3.88,  4.22  up 68+22:18:57    01:14:29
39 processes:  1 running, 38 sleeping
CPU: 29.0% user,  0.0% nice, 10.3% system,  3.1% interrupt, 57.6% idle
Mem: 1937M Active, 413M Inact, 291M Laundry, 2326M Wired, 1543M Buf, 330M Free
Swap: 

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
48908 mysql       665  20    0    13G    12G select   0  33.4H 106.79% mysqld`

Specifically do these lines

*Mem: 1937M Active, 413M Inact, 291M Laundry, 2326M Wired, 1543M Buf, 330M Free*
versus
48908 *mysql*       665  20    0    13G    *12G* select   0  33.4H 106.79% mysqld

indicate something off with the memory accounting in 'top' since mysql has 12G and the top line accounts for less than 6G?


----------



## Mjölnir (Aug 8, 2020)

Looks as if the accounting of shared memory is done as often as there are threads...  Consider to file in a bug report.  This AWS integration is fairly young, so there will more bugs than in more mature parts of the kernel.


----------



## gcomeau (Aug 10, 2020)

Is the presumption at the moment that this is not a VM issue but a Freebsd-aws port issue? And the bug surfaces using `top`?

I just want to properly route the report and see if I can package it in a smaller environment.


----------



## Mjölnir (Aug 10, 2020)

gcomeau said:


> Is the presumption at the moment that this is not a VM issue but a Freebsd-aws port issue? And the bug surfaces using `top`?


You can compare to the ouput of other tools, e.g. `procstat rusage <pid>`


----------

