# 4 hard disk 2 gmirror RAIDs performance problem



## ib2009 (Oct 10, 2009)

Hello,

I have a server with the following configuration: Supermicro 6015B-T+B, 32 Gb RAM, 4 Seagate SATA hard drives (ST3500320NS/SN05). FreeBSD 7.1-RELEASE-p8 amd64.

2 disks are configured as gm0 RAID-1 volume. 2 other disks are configured as gm1 RAID-1 volume. gm0 is for system and gm1 is for MySQL DB.

```
# mount
/dev/mirror/gm0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/gm0s1d on /tmp (ufs, local, soft-updates)
/dev/mirror/gm0s1f on /usr (ufs, local, soft-updates)
/dev/mirror/gm0s1e on /var (ufs, local, soft-updates)
/dev/mirror/gm1s1d on /usr/local/db (ufs, local, soft-updates)

# dd if=/dev/zero of=/1testfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 28.393003 secs (7386158 bytes/sec)

# dd if=/dev/zero of=/usr/local/db/1testfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 28.109289 secs (7460708 bytes/sec)
```
As you can see, the performance of disk subsystem is very bad.

I have another server with with FreeBSD 7.2 Release, same configuration, but only 2 hard disks:

```
# uname -a
FreeBSD sunzzz 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May 1 07:18:07 UTC 2009 [email]root@driscoll.cse.buffalo.edu[/email]:/usr/obj/usr/src/sys/GENERIC amd64
# mount
/dev/mirror/gm0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/gm0s1f on /tmp (ufs, local, soft-updates)
/dev/mirror/gm0s1d on /usr (ufs, local, soft-updates)
/dev/mirror/gm0s1g on /usr/local (ufs, local, soft-updates)
/dev/mirror/gm0s1e on /var (ufs, local, soft-updates)
/dev/md101 on /RAMDISK (ufs, local)
# dd if=/dev/zero of=/usr/local/db/1testfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 3.603078 secs (58204459 bytes/sec)
```
You can see, the second server hard disk speed about 58 Mbytes/sec, but the first server speed is only 7 Mbytes/sec.

How I can improve the hard disk performance of first server?


----------



## honk (Oct 11, 2009)

Is it not possible to use hardware RAID in this Supermicro server?


----------



## Carpetsmoker (Oct 11, 2009)

Maybe your RAID array is rebuilding? I've seem some people complain about poor performance while their RAID array was rebuilding (well, du'h).

Also, what options did you use with gmirror label/configure? Maybe this has an effect?

You can check both with gmirror list.


----------



## ib2009 (Oct 11, 2009)

It is possible to use the hardware RAID, but on all of my servers I use software RAID and actually don't see the reason to use hardware RAID.

Actually the both RAID are in the sync state. I used following commands to create the both RAIDs:


```
gmirror label -v -b round-robin gm0 /dev/ad4
gmirror insert gm0 ad6
gmirror label -v -b round-robin gm1 /dev/ad8
gmirror insert gm0 ad8
```


----------



## Carpetsmoker (Oct 11, 2009)

> Actually the both RAID are in the sync state.



Doesn't that mean that they're synchronizing? IIRC the COMPLETE state is used when they are synchronized.


----------



## aragon (Oct 11, 2009)

Yea, a synced gmirror volume looks something like this:


```
$ gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ad6s2
                      ad8s2
```


----------



## ib2009 (Oct 12, 2009)

There is a result:

```
# date && gmirror status && date && dd if=/dev/zero of=/usr/local/db/1testfile bs=1M count=200
Mon Oct 12 10:54:51 MSD 2009
      Name    Status  Components
mirror/gm0  COMPLETE  ad4
                      ad6
mirror/gm1  COMPLETE  ad8
                      ad10
Mon Oct 12 10:54:51 MSD 2009
200+0 records in
200+0 records out
209715200 bytes transferred in 29.882329 secs (7018034 bytes/sec)
```


----------



## aragon (Oct 12, 2009)

What speeds do you get when reading?

And how does it compare when you read from the drive devices directly?


----------



## ib2009 (Oct 12, 2009)

There are results:

```
# date && time cat /usr/local/db/1testfile > /dev/null && date && du -m  /usr/local/db/1testfile
Mon Oct 12 11:26:30 MSD 2009
        0.21 real         0.00 user         0.21 sys
Mon Oct 12 11:26:30 MSD 2009
201     /usr/local/db/1testfile

# date && dd if=/dev/mirror/gm1s1d of=/dev/null bs=1M count=1024 && date
Mon Oct 12 11:31:05 MSD 2009
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 47.382921 secs (22660946 bytes/sec)
Mon Oct 12 11:31:53 MSD 2009
```


----------



## aragon (Oct 12, 2009)

Try these rather:


```
dd if=/dev/mirror/gm1 of=/dev/null bs=1M count=1024
dd if=/dev/ad8 of=/dev/null bs=1M count=1024
```


----------



## ib2009 (Oct 12, 2009)

```
# dd if=/dev/ad8 of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 30.852072 secs (34802908 bytes/sec)

# dd if=/dev/mirror/gm1 of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 21.338481 secs (50319506 bytes/sec)
```


----------



## aragon (Oct 12, 2009)

Unless my eyes deceive me, it looks like your hard drives are just slow.  Looks like gmirror is actually speeding them up by reading from them in parallel.


----------



## DutchDaemon (Oct 12, 2009)

ib2009, will you please put 
	
	



```
tags around system output? Makes it so much more readable.
```


----------



## Carpetsmoker (Oct 12, 2009)

> Unless my eyes deceive me, it looks like your hard drives are just slow. Looks like gmirror is actually speeding them up by reading from them in parallel.



This is more than ``just slow''. My Quantum Fireball is faster. These drivers should do 50 or 60 MB/s _minimum_. Maximum would probably around 90 to 100 MB/s.

Some random thoughts that may help:

Are you sure your drives are correctly configured in your BIOS? i.e. You're not using ATA/33 or something? You can check with dmesg:
`% dmesg | grep ^ad`

Is your system actually doing anything? This should not affect performance this much, but you never know ... (tip: iostat(8)).

Can you (temporarily) delete one array? You can then test the drive without gmirror and see if gmirror is the culprit or something else.


----------



## ib2009 (Oct 12, 2009)

I think the disk are configured correctly, the problem is in the another place. Where? I don't know...


```
dmesg -a | grep ad
ad4: 476940MB <Seagate ST3500320NS SN05> at ata2-master SATA150
ad6: 476940MB <Seagate ST3500320NS SN05> at ata3-master SATA150
ad8: 476940MB <Seagate ST3500320NS SN05> at ata4-master SATA150
ad10: 476940MB <Seagate ST3500320NS SN05> at ata5-master SATA150
```

As I said above I have some Supermicro servers. More of them with 2 hdds (all servers with the Seagate hard disk). The normal speed of the ST3500320NS is about 100 Mbyte/sec. There is an example from another server:


```
# gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ad4
                      ad6
# sysctl -a | grep dev.ad
dev.ad.4.%desc: ST3500320NS/SN06
dev.ad.4.%driver: ad
dev.ad.4.%parent: ata2
dev.ad.6.%desc: ST3500320NS/SN06
dev.ad.6.%driver: ad
dev.ad.6.%parent: ata3
# dd if=/dev/zero of=/usr/local/1testfile bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes transferred in 2.162748 secs (96967013 bytes/sec)
```


----------



## ib2009 (Oct 12, 2009)

```
# iostat 10
      tty             ad4              ad6              ad8             cpu
 tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0  305 118.91  20  2.30  119.81  20  2.29  56.03  52  2.84   3  0  1  0 96
   0  418 14.27   3  0.04  14.27   3  0.04  16.21   8  0.12   1  0  1  0 98
   0  659  5.61   1  0.00   5.61   1  0.00  15.76   6  0.10   1  0  1  0 98
   0  795  0.00   0  0.00   0.00   0  0.00  19.54  54  1.03   1  0  0  0 99
   0  315 14.06   1  0.01  14.06   1  0.01  19.26  25  0.48   2  0  2  0 97
   0   87  1.14   1  0.00   1.14   1  0.00  21.19   7  0.14   2  0  1  0 98
   0   85  0.50   0  0.00   0.50   0  0.00  19.32  24  0.45   1  0  0  0 99
   0  737  6.70   0  0.00   6.70   0  0.00  19.23  46  0.87   1  0  0  0 99
   0  287  0.50   0  0.00   0.50   0  0.00  17.22  13  0.22   6  0  3  0 92
   0   94  1.40   0  0.00   1.40   0  0.00  18.55   4  0.07   2  0  2  0 96
   0   93  5.67   0  0.00   5.67   0  0.00  20.19  47  0.93   1  0  1  0 97
   0   94  9.80   0  0.00   9.80   0  0.00  18.61  25  0.45   1  0  0  0 99
   0  505  6.62   0  0.00   6.62   0  0.00  52.62   2  0.11   2  0  1  0 98
   0  556 12.56   2  0.02  12.56   2  0.02  20.08  39  0.77   1  0  1  0 99
   0  557  0.50   0  0.00   0.50   0  0.00  18.43  27  0.49   1  0  1  0 98
   0   94  5.67   0  0.00   5.67   0  0.00  30.52   2  0.06   1  0  1  0 99
   0  563  9.80   0  0.00   9.80   0  0.00  18.57  37  0.66   1  0  1  0 99
   0 1172  0.00   0  0.00   0.00   0  0.00  17.92  22  0.39   2  0  1  0 97
   0  805  0.00   0  0.00   0.00   0  0.00  21.04   7  0.15   2  0  2  0 97
   0  351  5.67   0  0.00   5.67   0  0.00  17.91  24  0.42   1  0  1  0 99
```

I can not delete one of arrays because 1 array for system and the second array for DB. My manager plan to migrate to another server, so I will do some experiments later.


----------



## terminus (Oct 20, 2009)

It is old problem with gmirror. There is two patches:

old:
http://www.freebsd.org/cgi/query-pr.cgi?pr=113885&cat=

and fresh one:
http://lists.freebsd.org/pipermail/freebsd-current/2009-April/006310.html


Unfortunately none of them is commited jet.


----------



## aragon (Oct 21, 2009)

terminus, I think you missed the part where it was shown that the hard drives themselves are slow...


----------

