# How to find out which process generates high disk usage



## tty23 (Aug 6, 2011)

Hi,

I have troubles with high disk usage on my FreeBSD system.

The system disk is used for 100% (according to gstat) for most
of the time. This is very annoying because the systems becomes
unresponsive.

It seems that this has something to do with sendmail. When I kill
all sendmail stuff and the periodic scripts the high disk usage
is gone.

These are some of the processes:

```
root    85293  0,0  0,0  8340   308  ??  IJ    3:02am   0:00,00 /bin/sh /etc/periodic/daily/440.status-mailq
smmsp   85294  0,0  0,0 36724  1628  ??  DJ    3:02am   1:15,95 mailq -Ac (sendmail)
root    85295  0,0  0,0  5828   280  ??  IJ    3:02am   0:00,00 tee /dev/stderr
root    85296  0,0  0,0  9120   268  ??  IJ    3:02am   0:00,00 egrep -v (mqueue is empty|Total requests)
root    85297  0,0  0,0  2764   200  ??  IJ    3:02am   0:00,00 wc -l
```

The funny thing is, I deleted 
	
	



```
/etc/periodic/daily/440.status-mailq
```
, still it is running.

Is there any way to figure out which process generates all the disk access? This would
be a great help to find out what is going on.

Thanks

Johannes


----------



## rob34 (Aug 6, 2011)

You could try running top and then pressing "m" to run in I/O mode.


----------



## tty23 (Aug 6, 2011)

That helps 

I guess I have to look out for high VCSW numbers.

The offending seem to be sendmail. Any ideas on what
could be the reason for sendmail to hijack my disk?

One other process, that looks strange is yarrow. Any
ideas on what this is?


----------



## jalla (Aug 6, 2011)

tty23 said:
			
		

> One other process, that looks strange is yarrow. Any
> ideas on what this is?



Could be this one ?


----------



## tty23 (Aug 6, 2011)

jalla said:
			
		

> Could be this one ?



Yay, nice find, this seems to be it:
http://en.wikipedia.org/wiki/Yarrow_algorithm

No needs to worry I guess 

I should have searched better before asking :r


----------



## SirDice (Aug 9, 2011)

tty23 said:
			
		

> The offending seem to be sendmail. Any ideas on what
> could be the reason for sendmail to hijack my disk?


A spammer using your system as a relay?


----------



## tty23 (Aug 10, 2011)

SirDice said:
			
		

> A spammer using your system as a relay?



Mhh, that is unlikely, it runs in my private network 

Here are the processes, that cause the high disk usage:

```
smmsp    1979  0,6  0,4 53108 15300  ??  DsJ  11:34am   1:20,31 sendmail: ./p77HRgWZ011358 from queue (sendmail)
smmsp    1980  0,6  0,4 53108 15344  ??  DsJ  11:34am   1:21,13 sendmail: ./p77HeFeh011358 from queue (sendmail)
```

I think these are the result mails from the "periodic daily" runs on my system. I guess for
some reason the mails are stuck in the queue.
What I do not understand:

Why are the mails not delivered?
Why do those undelivered mails saturate my OS disk with requests?

I guessed this entry from the maillog may be the cause:

```
Aug 10 06:35:26 frege sendmail[70019]: gethostbyaddr(IPv6:fc00:1::1) failed: 1
```

But I added an entry for this address to the hosts file, and still, the sendmail process
is stuck.

The output of  [CMD=""]service sendmail status[/CMD]:

```
Cannot 'status' sendmail. Set sendmail_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.
sendmail_submit is running as pid 22345.
sendmail_clientmqueue is running as pid 22347.
```

Any ideas?

I am a bit desperate :\


----------



## DutchDaemon (Aug 10, 2011)

What is inside these emails? Especially if you look in the qf* file, you should see the headers which may contain information about delivery problems. The df* file contains the body.


----------



## tty23 (Aug 10, 2011)

DutchDaemon said:
			
		

> What is inside these emails? Especially if you look in the qf* file, you should see the headers which may contain information about delivery problems. The df* file contains the body.



How to I figure out whats in the mails? Where should I look for the qf* and The df* files?


----------



## DutchDaemon (Aug 10, 2011)

Either in /var/spool/mqueue/ or in /var/spool/clientmqueue/. You can open these files with a regular text viewer like less.


----------



## tty23 (Aug 10, 2011)

I just noticed that the mails are not always the same, as I thought, but changing every second or so:

```
smmsp    1979  0,6  0,3 53108 11720  ??  DsJ  11:34am   1:57,74 sendmail: ./p6INdwu5042044 from queue (sendmail)
smmsp    1980  0,6  0,3 53108 11656  ??  DsJ  11:34am   1:58,85 sendmail: ./p77KeG6X011358 from queue (sendmail)
```


```
smmsp    1979  0,8  0,3 53108 10968  ??  DsJ  11:34am   1:58,29 sendmail: ./p6LL4XO3056500 from queue (sendmail)
smmsp    1980  0,8  0,3 53108 10920  ??  DsJ  11:34am   1:59,39 sendmail: ./p6MIbNog013507 from queue (sendmail)
```

and so on.

I just rechecked, `netstat -a` shows not strange connections.  Guess there is no spammer on my system.
However, I noticed the `DsJ` in the output of `ps aux`. So this is inside a jail.

The man page of `ps` tells me to look at `/proc/âŸ¨pidâŸ©/status` for details.
Yay, this is inside my "git" jail.

For all my jails, I have this in my /etc/rc.conf

```
# Prevent sendmail to try to connect to localhost
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
```

I thought, this would prevent `sendmail` from trying to send mails.

I guess this is wrong.

There are a bazillion mails in the /var/spool/clientmqueue on "git", even so many
that running `ls -l` does not finish within a few minutes.

EDIT1: 
There are 850324 mails :O. 
Note to self: `find` works much better in this case than `ls`.

This is what the mails contain:

```
This is a MIME-encapsulated message

--p6N1xVuZ008652.1311433795/git.xxx.xxx

The original message was received at Tue, 12 Jul 2011 18:17:37 +0200 (CEST)
from localhost
with id p69F3qBC087141

   ----- The following addresses had permanent fatal errors -----
postmaster
    (expanded from: postmaster)

   ----- Transcript of session follows -----
postmaster... Deferred: Connection refused by [127.0.0.1]
Message could not be delivered for 5 days
Message will be deleted from queue

--p6N1xVuZ008652.1311433795/git.xxx.xxx
Content-Type: message/delivery-status
```

The mail goes on and on, its about 56kBytes.

Any ideas how to stop that?

Why does the sendmail process even try to send mails, I thought
my config would prohibit this?


----------



## DutchDaemon (Aug 10, 2011)

Perhaps enabling outbound email to forward these nightly emails to an external address may help. See below for some insights. Note that a lot of programs and scripts are hardcoded to offer email to a local submission queue whether you like it or not.

http://lists.freebsd.org/pipermail/freebsd-questions/2006-April/120156.html
http://markmail.org/message/d2kspvd...ONE+page:1+mid:buyslvmzprdbjmjw+state:results

You can also alias all accounts mentioned in /etc/mail/aliases to /dev/null and run *newaliases*.


----------



## tty23 (Aug 10, 2011)

DutchDaemon said:
			
		

> Perhaps enabling outbound email to forward these nightly emails to an external address may help. See below for some insights. Note that a lot of programs and scripts are hardcoded to offer email to a local submission queue whether you like it or not.



Thanks for the hint, I will do that.


----------



## DutchDaemon (Aug 10, 2011)

Note: I'm not entirely sure if this works perfectly inside a jail, but on a system that only has to send out these system emails, you don't need any sendmail lines in /etc/rc.conf. None. The ones that are present in /etc/defaults/rc.conf produce exactly that result (i.e. Sendmail processing mails offered on localhost and sending them out). So putting in an alias to an external address (or /dev/null) should take care of the problem.


----------

