# Server keeps running out of RAM and uses swap even if inactive memory is very high



## Bergamin (Sep 28, 2020)

Hello guys,
I'm running a FreeBSD 12 server and I'm experiencing this strange issue. I've looked around in this forum for people with the same problem, but these topics where a bit too old and I think that the proposed solution might not apply to FreeBSD 12. Many where suggesting using
`sysctl vm.defer_swapspace_pageouts=1`
but on FreeBSD 12 is not possible (?).

My server has a total of 4GB of ram and 2GB of swap. My processes use about 3.6GB of ram. Right now, after a fresh restart, my server has about 900MB of free ram.
The problem is that after a while (let's say 1-2 days), inactive memory gets really high (up to 2GB) and swap usage goes really high too.
Why won't the server recycle the inactive memory?

I'm running a mysql+minecraft server on my machine. My guess (but I'm a newbie) is a mal configured mysql server. I left it with default settings, maybe I should try running mysqltuner.
A part from that, it seems to me very strange.

Has anyone run in a similar situation? Is there anything that I can to do force my server to recycle inactive memory?

Thanks in advance!


----------



## ekvz (Sep 28, 2020)

I've run into a similar situation where compiling a huge application would push my desktop machine into swap and left parts of the memory swapped out even long after the compilation was finished and there being multiple GB of available RAM again resulting in quite a bit of lag (i have a pretty slow harddisk) when applications tried to access it and it had to be paged in again. I've not investigated this much but the people i spoke to told me that there wasn't much (in the sense of kernel settings) i could do about it. I haven't checked the accuracy of this information yet but if there really isn't i might try to patch in some knobs myself. That would happen somewhere in the future though as right now i sadly don't have the time for it.


----------



## m0nkey_ (Sep 28, 2020)

For the Minecraft server, you might want to limit the RAM it is allowed to use with Java parms -Xms (initial RAM pool) and -Xmx (maximum RAM).. example: -Xms256M -Xmx2G, this will help in low memory environments, at least this was the fix a few years ago when I ran one. You also probably do need to tune your MySQL server to do the same thing, by limiting the RAM utilization for cached queries, etc.


----------



## mark_j (Sep 28, 2020)

Bergamin said:


> Why won't the server recycle the inactive memory?



Quite simply, why should it?
Inactive memory means it's... inactive. It relates to sleeping processes. It's highly likely that it's tagged to be swapped out should you require the memory.
If you want to remove inactive processes, start killing off stuff you're not using.

If your swap reaches its limit, then increase swap. See the handbook about how to do this (as a file): 11.12.2. Creating a Swap File

It might be a good idea to see if your mysql settings are overly generous.

I know nothing about your set up, but lots of things impact the memory consumption, from bloatware like web browsers to java, to certain desktops (looking at you KDE) and so on.

So, in summary, I'm saying it's too vague a question you're asking. However, for example, a headless server, 4GB RAM and 1/2 of that for swap is more than adequate. So, what's your setup and why is it exhausting virtual memory?


----------



## olli@ (Sep 28, 2020)

ekvz said:


> I've run into a similar situation where compiling a huge application would push my desktop machine into swap and left parts of the memory swapped out even long after the compilation was finished


That’s normal behavior. Why should the kernel waste time and I/O bandwidth swapping things in that are not (yet) used? It’s preferable to keep them in swap, in case something else needs the memory.

Older versions of FreeBSD even swapped out unused processes proactively after some time, even if there was sufficient free memory. Typical “victims” were the unused getty(8) process on the virtual syscons terminals; these always ended up in swap after some time. However, FreeBSD 12 doesn’t seem to do that anymore. I guess that feature was removed because today the size of RAM is typically much larger than the size of swap space. 25 years ago it was rather the opposite: swap space was usually larger than RAM.

Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is _nothing_ by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.


----------



## ekvz (Sep 28, 2020)

olli@ said:


> That’s normal behavior.



Oh, i didn't want to imply that this was an unreasonable default behavior in any way.



olli@ said:


> Why should the kernel waste time and I/O bandwidth swapping things in that are not (yet) used?



Because i hate having to wait ~30 seconds for my editors window to appear 



olli@ said:


> Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is _nothing_ by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.



I fully agree on this being the most pragmatic approach. Throwing money at the problem isn't always preferable though in which case having a bunch of settings to influence the swapping behavior would be pretty useful. Like i said before it's not like expect someone else to implement this for me and i'll probably just try to add some switches myself when i find the time. I guess i'll see how this works out when i get to it but i feel like it shouldn't be _that_ hard (i already have some kind of logic in mind that should stop it from becoming to trigger happy) and a little more flexibility wouldn't hurt, would it?


----------



## CyberCr33p (Sep 28, 2020)

Add this in /etc/sysctl.conf :


```
vm.pageout_update_period=0
```


----------



## Mjölnir (Sep 28, 2020)

`sysctl vm.defer_swapspace_pageouts=1`:I do not have this sysctl knob, do you mean `sysctl -d vm.disable_swapspace_pageouts`?


olli@ said:


> Today, the usual advice for people who have problems with things being swapped: Buy more RAM. 4 GB is _nothing_ by today’s standards, especially if your run Java + database stuff, and even more so if you put ZFS into the mix.


4 GB RAM should be fairly enough for a small DB; but 4GB RAM vs. 2 GB swap is a suboptimal choice.
Bergamin, please have look into this thread.  It boiled down to be an NFS issue.


----------



## CyberCr33p (Sep 28, 2020)

I use FreeBSD 12.1 :


```
sysctl -d vm.pageout_update_period
vm.pageout_update_period: Maximum active LRU update period

sysctl vm.pageout_update_period
vm.pageout_update_period: 600
```

Edit: I thought you were replying to me.


----------



## richardtoohey2 (Sep 28, 2020)

I'm working through something similar with MySQL 5.6 and mysqldump, but on a server with 32Gb of RAM.

A mysqldump script runs at 7.15 a.m., no swap is used.

*Exactly* the same script runs at 7.15 p.m., and mysqld uses swap and doesn't release it until the process is restarted.

So I _think_ something happening during the 7.15 p.m. run (MySQL busier?) that makes it/FreeBSD/jemalloc want to swap a bit.  I might have given InnoDB too much of the RAM.  Time to try mysqltuner myself!

More RAM should help your case but MySQL may still need some tuning (as in my case.)


----------

