# what is causing sbwait in php-fpm, how to debug



## gege (Oct 25, 2016)

Hi guys,

we upgraded server to FreeBSD 10.3-STABLE and php-fpm during high load of server started to make strange things (take all CPU, locks, sbwait, select etc). When server is not under load, everything works perfectly

top:

```
59 processes:  4 running, 55 sleeping
CPU: 11.7% user,  0.0% nice,  6.1% system,  0.4% interrupt, 81.8% idle
Mem: 3024M Active, 7578M Inact, 1956M Wired, 11M Cache, 1588M Buf, 3351M Free
Swap: 32G Total, 8016K Used, 32G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
63826 www           1  30    0   310M 45056K accept 19   0:06  14.70% php-fpm
63815 www           1  30    0   310M 45780K accept 21   0:06  13.77% php-fpm
63827 www           1  29    0   306M 42532K accept 14   0:06  13.77% php-fpm
63828 www           1  30    0   306M 42468K accept  9   0:05  13.28% php-fpm
63822 www           1  30    0   306M 42380K accept  5   0:06  13.18% php-fpm
...
 5085 www           1  23    0   118M 62504K kqread 18  44:06   6.49% lighttpd
763 root          1  34    0   298M 23264K kqread 11   0:30   0.00% php-fpm
11824 www           1  52    0   314M 38628K sbwait 11   0:22   0.00% php-fpm
13634 www           1  36    0   306M 35432K sbwait 21   0:21   0.00% php-fpm
14140 www           1  52    0   310M 35948K sbwait 14   0:19   0.00% php-fpm
13141 www           1  20    0   314M 35884K sbwait 18   0:18   0.00% php-fpm
11133 www           1  40    0   310M 36260K sbwait 19   0:17   0.00% php-fpm
...
```

What I dont like in this case is sbwait of php-fpm processes - as you can see, they are not respawned and they should be in accept state, I believe they get to this state after some timeout (?) to something. First they go to select state and end up in sbwait state taking 0% of CPU. I would like to debug and see what is causing of this sbwait (truss -p <PID> never finish) - please help me how I can debug this and find out for what they are actually waiting.

some values from php-fpm.conf

```
emergency_restart_threshold = 10
emergency_restart_interval = 1m
pm = dynamic
pm.max_children = 250
pm.start_servers = 30
pm.min_spare_servers = 10
pm.max_spare_servers = 30
pm.max_requests = 500
```

within php-fpm there are connections to various resources - mysql, memcache, redis, sphixsearch...but none of them giving timeout in logs.


----------



## gege (Oct 26, 2016)

to update this thread - we downgraded back to FreeBSD 10.1-STABLE and things get back to normal. No idea what was causing sbwait, attaching CPU usage

https://www.dropbox.com/s/8dhu54vdbbi79es/Screenshot 2016-10-27 02.57.57.png?dl=0


----------



## chrcol (Nov 5, 2016)

you need to diagnose this further as 10.1 soon wont be supported.

here is a quote from a FreeBSD developer.



> The sbwait wchan is present when a thread has invoked the in-kernel
> sbwait() function to wait for a socket event.  It's used in a number of
> situations, but the main ones are:
> 
> ...



So from that I would check into the following.

Increasing socket buffer sizes
Increasing open socket limits
Increasing socket queue limits, one that comes to mine is 'somaxconn' which has a very low default of 128
Check firewall configuration if the communication is over tcp.
Also ideally the php to mysql/memcache/redis etc. should be done over sockets rather than TCP.


----------



## CyberCr33p (Nov 8, 2016)

Does the latest version of FreeBSD 10-STABLE solve this issue? Also what php modules you use?


----------

