# Heavy load with apache22 (ufs state in `top`)



## balrog2000pl (Oct 18, 2009)

Hello!

After several minutes of operation my Apache is getting stuck and almost totally unresponsive and hanging on the ufs state greatly, as diagnosed by `top` tool.

I'd like to know what files exactly is it accessing at the moment it hangs ?

Is there such tool that will tell me. I tried `lsof` and `fstat` but i would like to see real-time reads/writes on the HDD.


Site is CakePHP + MySQL. Begins to operate normally and after some time it is getting stuck on ufs state.


----------



## DutchDaemon (Oct 18, 2009)

Try something like [cmd=]procstat -f -w 1 pid_of_httpd[/cmd] or run [cmd=]top -U www -m io[/cmd] to see disk activity. Assuming 'www' is your Apache user.


----------



## ale (Oct 18, 2009)

You can also try running ktrace.


----------



## balrog2000pl (Oct 18, 2009)

Thanks, these are quite useful. In the meantime I enabled noatime flag on /home and /var, seems to have helped a bit. 

Any more details on successfully using ktrace?

Regards


----------



## ale (Oct 18, 2009)

Run [cmd=]ktrace -f ktrace.out -p pid_of_httpd &[/cmd] and let it run for a while (not too much as it produces huge output).
Then run [cmd=]ktrace -C[/cmd] to stop tracing the process.
Now you can examine the output with [cmd=]kdump -f ktrace.out | less[/cmd]


----------



## DutchDaemon (Oct 18, 2009)

I think ktrace(1) and kdump(1) (and/or ale) tell you all you need to know. Be sure to have [cmd=]ktrace -C[/cmd] ready


----------



## ale (Oct 18, 2009)

Another option could be truss(1).


----------



## balrog2000pl (Oct 18, 2009)

Thanks DutchDaemon and ale!

ktrace showed me where is the problem. I have set session lifetime for quite long time (1 day), now the site is heavily visited, so after a day the directory contained 100k session files! 

Somehow and somewhen apache or httpd was listing the directory and, whats worse, stating each of 100k files :| 

Now I should probably find another session storage not to kill my server anymore.

Imagine stating 100k files concurrently by 10 processes, no wonder I had problems.


----------



## bestwc (Oct 19, 2009)

Tools?  vmstate?

Have you try to retrieve the log file?


----------



## tuxick (Sep 3, 2010)

*solution*

I ran into same, in our case it the solution was dirhashing:

`sysctl -a | grep dirhash`

will show you current state, and most likely increasing vfs.ufs.dirhash_maxmem will solve your problems.


----------



## User23 (Sep 3, 2010)

```
systat -vm
```
 gives you a general overview what happens on your disks. If you have thousands of directories and "vfs.ufs.dirhash_mem" is nearly as big as "vfs.ufs.dirhash_maxmem" then increasing "vfs.ufs.dirhash_maxmem" will help a lot. On a mail server i have set it to 30MB and it consumes nearly all of it


----------

