# Performance problem FreeBSD9.0 + ZFS



## cr4sh (Apr 10, 2013)

Hello, I have performance problem on one of my production server. I have it now 3 days, and can't solve or find where the problem is.

Hardware: HP ML350 G5, Smart Array E200i with 2 x 64SAS disk, 6GB RAM

System: FreeBSD hydra 9.0-STABLE FreeBSD 9.0-STABLE #1: Mon Apr  9 19:15:53 CEST 2012     cr4sh@hydra:/sys/amd64/compile/Hydra  amd64

/boot/loader.conf

```
vfs.zfs.prefetch_disable=1
vfs.zfs.cache_flush_disable=1
```

[cmd=]zfs list[/cmd]:

```
hydra# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      58,6G  4,40G   976M  legacy
zroot/jails                54,1G  4,40G  1,78G  /usr/jails
zroot/jails/dns             416M  4,40G   691M  /usr/jails/dns
zroot/jails/jailbase        317M  4,40G   314M  /usr/jails/jailbase
zroot/jails/mta            32,7G  4,40G  32,4G  /usr/jails/mta
zroot/jails/mysql55         897M  4,40G   851M  /usr/jails/mysql55
zroot/jails/postgresql9     625M  4,40G   669M  /usr/jails/postgresql9
zroot/jails/samba          3,04G  4,40G  3,25G  /usr/jails/samba
zroot/jails/tomcat7        1,01G  4,40G  1,05G  /usr/jails/tomcat7
zroot/jails/windowsXP      13,2G  4,40G  12,8G  /usr/jails/windowsXP
zroot/tmp                   107K  4,40G    73K  /tmp
zroot/usr                  3,33G  4,40G  2,27G  /usr
zroot/usr/home               46K  4,40G    46K  /usr/home
zroot/usr/ports            1,06G  4,40G   965M  /usr/ports
zroot/usr/ports/distfiles   102M  4,40G   102M  /usr/ports/distfiles
zroot/usr/ports/packages   14,0M  4,40G  14,0M  /usr/ports/packages
zroot/var                   226M  4,40G  75,1M  /var
zroot/var/crash            31,5K  4,40G  31,5K  /var/crash
zroot/var/db                136M  4,40G   135M  /var/db
zroot/var/db/pkg            778K  4,40G   778K  /var/db/pkg
zroot/var/empty              31K  4,40G    31K  /var/empty
zroot/var/log              1,66M  4,40G  1,55M  /var/log
zroot/var/mail              428K  4,40G   428K  /var/mail
zroot/var/run                87K  4,40G    64K  /var/run
zroot/var/tmp                32K  4,40G    32K  /var/tmp
```

[cmd=]jls[/cmd]

```
hydra# jls
   JID  IP Address      Hostname                      Path
     1  192.168.34.10   mx                    /usr/jails/mta
     2  192.168.34.16   dns                   /usr/jails/dns
     3  192.168.34.8    tomcat7               /usr/jails/tomcat7
     4  192.168.34.9    postgresql9           /usr/jails/postgresql9
     5  192.168.34.12   mysql                 /usr/jails/mysql55
     6  192.168.34.11   xp                    /usr/jails/windowsXP
     7  192.168.34.15   samba                 /usr/jails/samba
```

Issue:
Sometimes I get server lagging so much I can't use it on console.. When i type [cmd=]w -n[/cmd] I need to wait about 20-30 sec. then I get logged users. I think fold was jail xp (nginx + php) but one or two time I get laggy without this jail.

[cmd=]vmstat[/cmd] show nothing, in [cmd=]dmesg[/cmd], messages are any errors. I try memtest memory, but it passes all tests.

I check [CMD=]dd if=/dev/zero of=/tmp/test bs=1m count=100[/cmd] 
Sometimes I get this:

```
hydra# dd if=/dev/zero of=/tmp/test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.599228 secs (174987776 bytes/sec)
```
And sometimes this:

```
hydra# dd if=/dev/zero of=/tmp/test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 26.476963 secs (3960333 bytes/sec)
```

I don't have other ideas. I will be grateful for any help.

Best regards.


----------



## ta0kira (Apr 10, 2013)

Are you running low on memory? When I first set up ZFS it would eventually take up ~10GB of memory, which would remain locked for eternity. I can imagine that having as many pools as you do could cause it to take up nearly all of your memory. I limited ARC and total kernel memory and that took care of the problem. Given 16GB of RAM:
	
	



```
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="512M"
vfs.zfs.prefetch_disable="1"
```
Other values might be appropriate, but I just needed to set limits and those seem to work fine. This page might also be helpful.

If your problem isn't memory, maybe you can give the jails a higher nice or give your own login a lower nice (e.g. :priority=-5:) in login.conf.

Kevin Barry


----------



## Sebulon (Apr 10, 2013)

Hey,

@cr4sh

uncomment the entries in loader.conf, you donÂ´t seem to really know what they do any way, considering that you have disabled cache flushing without a proper SLOG. That is outright dangerous.


@ta0kira

By limiting kmem_size_max, you have effectively also limited what all of your other applications could be using. The only tunable you may keep is arc_max, but raise it. With 16GB of RAM, about 15 is really available, consider what other applications you have and make a rough guess how much RAM theyÂ´d like, lets say another 4, so you take 15-4=11 should end you up with vfs.zfs.arc_max="11G". Comment out the rest and see your performance shoot through the roof.

/Sebulon


----------



## kpa (Apr 10, 2013)

This has repeated many times but do not touch kmem_size_max, IT IS NOT WHAT YOU THINK IT IS so leave it alone and don't set it at all. The same goes for most VM and ZFS related tunables on AMD64. The sole exception is the above mentioned vfs.zfs.arc_max.


----------



## rusty (Apr 10, 2013)

Does the 80% rule for ZFS still stand?
If so, it'll be a contributing factor in your setup.


----------



## cr4sh (Apr 10, 2013)

Hello.
My server is now laggy. 
About that loader.conf i add this after my server get problems. I can comment it, but this dont help me.

Load :


> 0,00 0,01 0,00



Top show :


> last pid: 18567;  load averages:  0.02,  0.04,  0.01                                                                up 0+07:42:31  08:19:23
> 203 processes: 1 running, 202 sleeping
> CPU:  0.2% user,  0.0% nice,  1.2% system,  0.1% interrupt, 98.5% idle
> Mem: 629M Active, 377M Inact, 4695M Wired, 6640K Cache, 221M Free
> Swap: 4096M Total, 11M Used, 4085M Free



gstatus show :


> 10 301 18 86 14.4 2250 1410 24.2 100.08| da0
> 10 301 18 86 14.4 2250 1410 24.2 100.08| da0p3


----------



## Sebulon (Apr 10, 2013)

@cr4sh

So itÂ´s not memory related. By the looks of it your disks are working their ass off (probably with your databases). Buy a really good SSD, like D2RSTK251E19-0200 e.g. Make sure to partition it aligned to 1MiB and do proper IO-sizing with gnop. Then add it to your pool as a log device. That is going to offload your disks completely.

/Sebulon


----------



## Crivens (Apr 10, 2013)

It looks like you have the 2 discs attached to the "smart" controller and let that controller do some raid, yes?

The bottleneck seems to be exactly that controller (100.08% load, if I read the last post correctly), so no tuning in ZFS can get that corrected. You might consider to add the disks directly, and skip the controller software, because ZFS does it's own raid management.


----------



## cr4sh (Apr 10, 2013)

@Crivens 
You have right. ZFS is on RAID0 from E200i. But for now all works fine. I dont add new services or smt. And it crash ;/
I can buy for now ServeRAID M1015 LSI SAS2008 but i dont know how it works with 4K sector disks...


----------



## Sebulon (Apr 10, 2013)

cr4sh said:
			
		

> I can buy for now ServeRAID M1015 LSI SAS2008 but i dont know how it works with 4K sector disks...



The SAS2008 chip handles 4k disks just fine. Just flash the card with the standard LSI Initiator/Target(IT) firmware to "dumb" it down to a standard HBA, from what I have read about it.

/Sebulon


----------



## cr4sh (Apr 10, 2013)

@Sebulon 
Thanks for info. 

If anyone have more idea with my case, i will be greatfull


----------



## r_t_f_m (Apr 10, 2013)

Your hard drives are starving, looks like your files are fragmented 
(using database (MySQL p.ex) on ZFS is often the issue).

Try to stop all jails, and check with [CMD="gstat"] -a[/CMD] that 
there is no or very low disk activity before testing the raw disk speed 
(just to exclude any issue with the controler)
 [CMD="dd "] if=/dev/da0 of=/deb/null bs=1m [/CMD] and check the speed.

You are also very low on free space 4,4GB/64GB, when writing or 
modifying files, ZFS is seeking free blocks on hard drive which 
could explain trashing (busy 100%).

I've had similar issue on my production server, MySQL on ZFS, after a long 
time of fighting I've moved all MySQL files to UFS.


----------



## cr4sh (Apr 10, 2013)

Im back with my new controller  
I reflash it to IT flash.
I test [CMD=""]dd bs=1m count=1000[/CMD] from da0 and to da0 i got ~100-120M that is not bad i think.

Thanks all for help.


----------



## Sebulon (Apr 10, 2013)

@cr4sh

Wow, that was fast! Although the test you performed only measures a continous, big read/write, and the performance issues you are probably experiencing are lots of small random writes generated from your databases. ThatÂ´s why I think what your system really needs is a log device in the pool to use as a write cache for all those sync-writes. The log device also protects the data in the pool from being fragmented:

ZFS Fragmentation

/Sebulon


----------

