# Out of swap space?



## NewGuy (Nov 21, 2014)

Recently I've been finding my MySQL server has been crashing about once a day. When I looked through the log files I found the following message which appears to indicate why mysqld was stopped.


```
Nov 21 12:22:41 myhost kernel: sonewconn: pcb 0xfffff800111a9310: Listen queue overflow: 193 already in queue awaiting acceptance
Nov 21 12:23:06 myhost last message repeated 5459 times
Nov 21 12:24:30 myhost kernel: swap_pager: out of swap space
Nov 21 12:24:30 myhost kernel: swap_pager_getswapspace(16): failed
Nov 21 12:24:30 myhost kernel: pid 1078 (mysqld), uid 88, was killed: out of swap space
Nov 21 12:24:46 myhost kernel: swap_pager_getswapspace(16): failed
```

The "out of swap space" error would suggest I'm running low on memory. However, whenever I look at `top` I see I've got some free space left, usually around 50MB, and a few hundred MB is "unused" memory. I have about 2GB of swap space and only 20MB is being used. So it seems unlikely that 2GB of swap are being taken all at once.

So I wonder if the real issue here is sonewconn with its overflowing queue. I found this mailing list post (http://lists.freebsd.org/pipermail/freebsd-net/2013-July/036151.html) saying this was a known issue (probably from a DDoS attack) and a fix was in the works. Any thoughts on how I can work around this in the mean time?


----------



## junovitch@ (Nov 22, 2014)

It's not necessarily an external DDoS and could very well be an internal application doing the DoS if it is squandering resources.  A while back, there was a buggy OpenJDK version that never closed file descriptors.  Eventually my media server would touch too many files as the limit was about 10,000 for FreeBSD 9 and I would see the same sonewconn messages.

If you can run and post the results that would be helpful.
`swapinfo`
`sysctl kern.maxfiles`
`sysctl kern.maxfilesperproc`
`sysctl kern.openfiles`
`sysctl kern.ipc.maxsockets`
`sysctl kern.ipc.numopensockets`
`sysctl kern.ipc.soacceptqueue`

Additionally, this would probably be helpful.
`freebsd-version; uname -a`


----------



## NewGuy (Nov 22, 2014)

junovitch said:


> It's not necessarily an external DDoS and could very well be an internal application doing the DoS if it is squandering resources.  A while back, there was a buggy OpenJDK version that never close file descriptors.  Eventually my media server would touch too many files as the limit was about 10,000 for FreeBSD 9 and I would see the same sonewconn messages.
> 
> If you can run and post the results that would be helpful.
> `swapinfo`
> ...



I know it was a DoS attack because my web server logs show a handful of remote clients hitting the server over and over, all loading several pages per second. This seems to happen early in the morning most mornings. I added a filter to the firewall to throttle connections from individual clients and the problem has gone away.

My host is running FreeBSD 10.0 (64-bit)

`swapinfo`

```
Device          1K-blocks     Used    Avail Capacity
/dev/label/swap0.eli   2047996    61020  1986976     3%
```


```
maxfiles: 49312
maxfilesperproc: 28755
openfiles: 649
maxsockets: 31955
numopensockets: 167
soacceptqueue: 128
```


----------



## junovitch@ (Nov 22, 2014)

Good deal that you got to the bottom of it.  You may want to look at enabling accf_http(9) and accf_data(9) for HTTP and HTTPS, respectively.  That may assist in keeping load on your web server low when it is under heavy load.  What web server are you using?


----------



## NewGuy (Nov 23, 2014)

junovitch said:


> Good deal that you got to the bottom of it.  You may want to look at enabling accf_http(9) and accf_data(9) for HTTP and HTTPS, respectively.  That may assist in keeping load on your web server low when it is under heavy load.  What web server are you using?



I will look into that, thank you. I am running Apache web server, version 2.4. It's a standard AMP stack for a small site I host.


----------



## junovitch@ (Nov 23, 2014)

I'm not an Apache guy but looking at the Apache RC script it looks like setting an  rc.conf option should do it.  If you run across Nginx just look up the config file manual online as it's a config file option.

```
apache24_http_accept_enable="YES"
```

If you run it in a jail, you'll have to load the kernel modules manually from the host so the jail can use them.

`echo 'accf_http_load="YES"' >> /boot/loader.conf`
`echo 'accf_data_load="YES"' >> /boot/loader.conf`
`kldload accf_http`
`kldload accf_data`


----------



## NewGuy (Nov 23, 2014)

Thank you, I will try running with accf_http enabled and see how that works. So far putting a rule in my PF configuration file has worked well, but I think this might help reduce the work my host is doing too.


----------



## SirDice (Nov 24, 2014)

You also want to make sure your MySQL server isn't accessible from the internet. Bind it to 127.0.0.1 if you have to. There's very little access logging with MySQL, you could be hammered from all directions without knowing it. Most MySQL servers are only used from a web application so binding it to 127.0.0.1 should be enough to protect it from any outside attacks.


----------



## NewGuy (Nov 24, 2014)

SirDice said:


> You also want to make sure your MySQL server isn't accessible from the internet. Bind it to 127.0.0.1 if you have to. There's very little access logging with MySQL, you could be hammered from all directions without knowing it. Most MySQL servers are only used from a web application so binding it to 127.0.0.1 should be enough to protect it from any outside attacks.



I agree. Fortunately, I have MySQL behind the firewall so nothing is coming in from the net, my connections to the database are local.


----------

