# out of swap space



## tarkhil (Apr 11, 2021)

```
Apr 11 21:00:41 bipgame kernel: pid 10470 (mysqld), jid 2, uid 88, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 31041 (bhyve), jid 0, uid 0, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 42311 (node), jid 7, uid 0, was killed: out of swap space
Apr 11 21:00:46 bipgame kernel: pid 61686 (php), jid 20, uid 1001, was killed: out of swap space
```
this disaster hits me from time to time, and I'm asking for advice.


I do have lots of swap space. Monitoring does not show growing memory usage. 
It hits sometimes between 00:30 and 01:50 of an hour, about 2-3 times a week. No, nothing suspicious in crontabs.
What can I check and/or tune? FreeBSD 13.0-RC5 (which is almost 13.0-RELEASE)


----------



## ct85711 (Apr 11, 2021)

I recall from several threads in regards to mysql frequently consuming a lot of memory, more specifically from when it goes and backups the database(s) and or database dumps.  By any chance you have any jobs that is scheduled to backup the system and/or the database?


----------



## VladiBG (Apr 11, 2021)

Which malloc-lib you are using for MySQL?


----------



## richardtoohey2 (Apr 12, 2021)

It will be a bit sad if it is MySQL.  My _brief_ testing with one of the 13.0 BETA/RC releases _seemed_ to handle the MySQL import a lot better.

I've also noticed long-running rsync processes dipping into swap.

As for advice - I have this running every minute to try and nail down what processes are doing what - to try and find a pattern:

```
@every_minute date >> /tmp/swap_usage.txt && top -n 100 >> /tmp/swap_usage.txt && pstat -sh >> /tmp/swap_usage.txt
```

Keep an eye on the tmp file and if you are lucky you may find a pattern.

A minute is eons in terms of what can happen on a modern machine, but hopefully it might give you some things that are worth investigating further.


----------



## tarkhil (Apr 12, 2021)

VladiBG said:


> Which malloc-lib you are using for MySQL?


default one. mysql57-client-5.7.31_1 (something in application fails with mysql8). Any ideas on changing malloc?


----------



## SirDice (Apr 12, 2021)

Are you running ZFS in combination with bhyve and MySQL? Then make sure you limit `vfs.zfs.arc_max`. Also check with a tool like databases/mysqltuner that your MySQL server isn't using too much memory. And off course double check if ARC + bhyve memory + MySQL memory usages don't use a lot more than your machine actually has.


----------



## VladiBG (Apr 12, 2021)

Solved - FreeBSD 12.x and MySQL 5.7 and importing file with lots of small lines exhaust RAM and swap
					

Thought I'd try out the new 12.1 beta and RC releases but have got side-tracked down a MySQL 5.7 rabbit hole.  When I try and import a 17Gb database dump file from mysqldump my machines run out of RAM then they swap, then they run out of that, then the OOM killer kicks in with sad results...




					forums.freebsd.org


----------



## tarkhil (Apr 12, 2021)

SirDice said:


> Are you running ZFS in combination with bhyve and MySQL? Then make sure you limit `vfs.zfs.arc_max`. Also check with a tool like databases/mysqltuner that your MySQL server isn't using too much memory. And off course double check if ARC + bhyve memory + MySQL memory usages don't use a lot more than your machine actually has.


zfs, and bhyve (but mysql is in jail). Mysqltuner claims that
[OK] Maximum possible memory usage: 4.6G (7.22% of installed RAM)

so it cannot request THAT much memory. 

Okay, I'll think about limiting ARC, but it was not a problem for quite a time.


----------



## SirDice (Apr 12, 2021)

Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough, so you get a battle between MySQL, bhyve and ARC all trying to grab the same memory. That usually doesn't end well. Tweaking ARC usually helps.


----------



## tarkhil (Apr 12, 2021)

I'll do


SirDice said:


> Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough, so you get a battle between MySQL, bhyve and ARC all trying to grab the same memory. That usually doesn't end well. Tweaking ARC usually helps.



but isn't laundry too big?

```
Mem: 17G Active, 2347M Inact, 21G Laundry, 15G Wired, 2340M Free
ARC: 7896M Total, 1061M MFU, 4808M MRU, 54M Anon, 202M Header, 1717M Other
     4579M Compressed, 6112M Uncompressed, 1,33:1 Ratio
Swap: 128G Total, 13G Used, 115G Free, 9% Inuse, 108K In
```


----------



## VladiBG (Apr 12, 2021)

You can use `top` to see per process swap usage.


----------



## tarkhil (Apr 12, 2021)

VladiBG said:


> You can use `top` to see per process swap usage.


Overall free swap is tens of gigabytes. About 100Gb.


----------



## VladiBG (Apr 12, 2021)

Did you find which process took all your memory and start using the swap?


----------



## SirDice (Apr 12, 2021)

Agree, there was at some point an application that used up all memory, it's usually not the ones that get killed, those are typically innocent bystanders. Figure out which application or process got hammered. Maybe a run-away web process? Besides ZFS, MySQL, bhyve, node and PHP what else is running on that machine?


----------



## zirias@ (Apr 12, 2021)

SirDice said:


> Definitely try to limit ARC. Apparently it can get a bit lazy and not release memory quick enough


This is definitely much better on 13, though – I've seen ARC drop by several GB from one second to the next. Maybe there are still situations that require limiting it, but I'd observe this a bit first.


----------



## richardtoohey2 (Apr 12, 2021)

Did you try my cron suggestion?  Leave it running for 24-48 hours and see if you can see any pattern.

MySQL is my problem child (on 11.x and 12.x certainly) - I've had to make the %age of RAM smaller and smaller and smaller to make it play nicely with Apache and PHP.  I don't use ZFS so ARC not an issue in my case.  I think 4-5GB max allocated to MySQL on a 32GB RAM machine - even though plenty of recommendations say 25% or 50% RAM (obviously those depend on what else is running on the same machine).


----------



## SirDice (Apr 13, 2021)

richardtoohey2 said:


> I think 4-5GB max allocated to MySQL on a 32GB RAM machine


You should look at that size of the data. Adding too much memory to MySQL is bad for its performance, it'll spend more time managing caches and buffers than actually using it. Ideally you want to assign a little more than the data to allow for some growth. But don't overdo it.


----------



## wdick (Apr 13, 2021)

For me it was rkhunter run through periodic, which killed processes with out of swap memory.

The actual problem is `lsof` with this bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250929

See: https://forums.freebsd.org/threads/...-out-of-swap-space-message.78653/#post-493901


----------



## grahamperrin@ (Oct 10, 2021)

tarkhil if this is still an issue, <https://forums.FreeBSD.org/threads/81380/post-536020> mentions two ARC-related sysctls that may be of interest.


----------



## Alain De Vos (Oct 10, 2021)

Check the output of:

```
ps axvmw | head
```
Verify mysql innodb_buffer_pool_size not too large.


----------

