# System instability, 10.3



## Hornpipe2 (Aug 17, 2016)

I've got a server with Pentium 4, 512MB RAM, 2gb swap, a spinning HDD with plenty of space and a 100mbps NIC.

Things seemed to be working okay before, but something I've done has led to system instability.  After a reboot things are good for a few days, then the web server shuts down.  After a while I notice it, log in and run dmesg, and get these:


```
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
...
vm_thread_new: kstack allocation failed
vm_thread_new: kstack allocation failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc3f0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc7e0: pru_attach() failed
sonewconn: pcb 0xc52bc5e8: pru_attach() failed
...
```

I have no idea how to go about troubleshooting an issue like this, or even what it means.  Can someone help me walk myself through this to figure out the problem?

EDIT: Looking into /var/log/messages, here is some useful info.

```
Aug 12 18:38:54 server ntpd[690]: leapsecond file ('/var/db/ntpd.leap-seconds.li
st'): good hash signature
Aug 12 18:38:54 server ntpd[690]: leapsecond file ('/var/db/ntpd.leap-seconds.li
st'): loaded, expire=2016-06-01T00:00:00Z last=2015-07-01T00:00:00Z ofs=36
Aug 12 18:38:54 server ntpd[690]: leapsecond file ('/var/db/ntpd.leap-seconds.li
st'): expired less than 73 days ago
Aug 13 03:18:14 server kernel: vm_thread_new: kstack allocation failed
Aug 13 03:55:51 server kernel: vm_thread_new: kstack allocation failed
Aug 13 03:55:52 server sshd[1500]: error: fork: Cannot allocate memory
Aug 13 04:29:18 server kernel: vm_thread_new: kstack allocation failed
Aug 13 04:29:22 server last message repeated 4 times
Aug 13 04:29:19 server sshd[1500]: error: fork: Cannot allocate memory
Aug 13 04:29:22 server last message repeated 4 times
Aug 13 04:29:23 server kernel: vm_thread_new: kstack allocation failed
Aug 13 04:29:23 server sshd[1500]: error: fork: Cannot allocate memory
Aug 13 04:29:24 server kernel: vm_thread_new: kstack allocation failed
Aug 13 04:29:24 server sshd[1500]: error: fork: Cannot allocate memory
Aug 13 04:29:25 server kernel: vm_thread_new: kstack allocation failed
Aug 13 04:29:25 server sshd[1500]: error: fork: Cannot allocate memory
Aug 13 04:29:26 server kernel: vm_thread_new: kstack allocation failed
```

So it appears that sshd cannot start because the server runs out of RAM.  I haven't been having this problem before recently, though.  Why would I be running out now?


----------



## fossette (Aug 17, 2016)

top(1) might be your best friend.  Use the `o` interactive command to change the default sort order and specify _*size*_ (total size of the process) or _*res*_ (current amount of resident memory).  Check the `man` page!  Lots of options for this little gem.


----------

