# lockf state - php - lock contention?



## chrcol (Jan 9, 2012)

Hi, I need to know what lockf is and how to diagnose high system cpu usage. The problem is as follows. The server is a web server, it has no mysqld and it only serves dynamic files no static media.

When I was using apache with mod_php and load got high enough then all http processes would enter lockf state and the entire http server remained locked until forcefully restarted.  When this occured cpu usage was quite low and there was no obvious resource saturation cpu, i/o etc.  Enabling vfs shared lookups eased the problem but didn't solve it.

I have now changed apache to threaded and separated php off to a fastcgi setup.  I have tested both fcgid and fpm.

Now http itself has no issues so it's the php side of the serving that's the problem.  But I get the same issue, when enough connections to server system% load jumps up in one go from about 10% to 90% and then all processes lockf state.

I suspect something in the web code itself is doing something freebsd FreeBSD doesn't like however I don't know how to diagnose it from here as I cannot see the following.

1 - breakdown of whats using sys% cpu usage.
2 - live status of files accessed on HDDs.

Here is a snapshot from *systat*, I even have /tmp on a RAM disk as it was been flooded with php session files.


```
2 users    Load  3.12  2.76  2.24                  Jan  9 06:25

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act 1604080    8560  7193324    11168 5817080  count
All 1726784   12964 1084943k   139136          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt    144 cow    1616 total
  1          1k       507 1759 1388   16  117 1668   1023 zfod        atkbd0 1
                                                          ozfod       ata0 irq14
74.5%Sys   0.0%Intr  0.9%User  0.0%Nice 24.6%Idle        %ozfod       uhci4 22
|    |    |    |    |    |    |    |    |    |    |       daefr   400 cpu0: time
=====================================>                    prcfr       ciss0 256
                                        19 dtbuf          totfr    16 bce1 258
Namei     Name-cache   Dir-cache    206497 desvn          react   400 cpu2: time
   Calls    hits   %    hits   %     36777 numvn          pdwak   400 cpu3: time
    2319    2319 100                 33420 frevn          pdpgs   400 cpu1: time
                                                          intrn
Disks   md0   da0 pass0                            634752 wire
KB/t   0.00  0.00  0.00                           1566956 act
tps       0     0     0                             83288 inact
MB/s   0.00  0.00  0.00                               236 cache
%busy     0     0     0                           5817032 free
```

So only 74% sys but that's enough, currently all php-fpm in lockf state and either not responding or responding very slowly, if I forcefully restart it (php-fpm) then it instantly becomes responsive again with sys% down to around 10%.

The hardware is 4 SCSI HDDs in RAID10, 4 core Intel Xeon.  FreeBSD 8.2 64bit generic kernel.


----------



## chrcol (Jan 19, 2012)

Well I did the only test that seemed logical which was to try the same configuration, same web files on another OS.

On FreeBSD the load slowly increases as connections increase then will suddenly topple over with mass lockf and high system cpu usage, on debian no such thing happens and its happily dealing with many multiples of the load.  The top over point on FreeBSD is not the same every time its highly variable.  The servers are all high spec and its occuring on 7.x,8.x and 9.0.  As no one here seems to not be able to help or willing to help it seems on FreeBSD I am at a dead end to get around the issue.  The only references to lockf I could really find were people binding apache to specific ips which isnt applicable here as (a) I changed apache to * listen on all ips and (b) its happening on php-fpm a different application.  Cant say I am happy with what I have found as I like FreeBSD but I have to use what works so for this particular site I will have to change unless a solution gets found but of course once I change and it stays working well they are unlikely to accept a change back regardless if it gets fixed or not.

If a developer or someone else with an idea wants to test I can keep a server up with the data on.


----------



## CyberCr33p (Mar 1, 2019)

I know this is an old thread but did you find anything else about the lockf issue ?


----------

