# Some performance tests with zfs and different raidtypes and sizes



## jalla (May 31, 2012)

Some time ago I did some testing with bonnie++ of various RAID types with varying sizes of vdevs. As my test environment was a bit awk*w*ard, the measured performance probably doesn't mean much to anybody, but I think it's valid in showing relative performance of the different scenarios.

The disks are 750Gb Seagate SATA (fibre-channel), probably 5-6 years old. The fibre-channel controller is Qlogic 2432 (PCIe)

A few comments on the numbers in general.

The write speed to these drives is obviously rotten. My best guess is it's because they have a custom firmware, and are probably optimized for the specialized storage system I nicked them from.

The disks are connected on a 2Gbit fibre-channel link, so maximum aggregate throughput is limited to 250Mb/sec.

Tests are named Rn-DxV, which is RAID type, number_of_disks, and number_of_vdevs, respectively (r1-10x5 is 5 pairs of mirrors, etc).

```
-------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
r0-1x1   16384 12452  3.5 12457  0.7  9701  4.1 69571 22.8 76289  5.0 435.6  0.8
r0-3x1   16384 30372 10.5 24044  2.8 22521  3.2 224798 63.4 218721  8.2 490.2  0.6
r0-6x1   16384 61981 21.2 46135  5.4 38096  5.2 241451 68.0 235960  9.9 582.9  0.7

r1-2x1   16384  7827  2.7  6443  0.8  6098  0.9 114324 32.6 133370  5.5 223.3  0.4
r1-4x2   16384 18321  4.5 14529  1.7 13860  2.0 237875 69.2 206736  7.0 434.6  0.6
r1-6x3   16384 30104  7.4 23427  2.7 21700  3.1 237144 74.8 221707  7.6 468.3  0.6
r1-8x4   16384 40132  9.8 31087  3.6 27123  3.9 231847 72.9 232825  8.2 605.7  0.7
r1-10x5  16384 50620 12.3 36697  4.2 33167  4.7 242218 75.4 216236  8.3 520.6  0.7
r1-12x6  16384 58827 20.2 44535  5.5 36431  5.2 248551 70.0 225497  9.4 560.6  0.7

r1-3x1   16384  7811  2.7  6474  0.7  6065  0.9 203852 57.8 181481  7.1 256.4  0.3
r1-4x1   16384  7771  2.7  6406  0.7  6111  0.9 220671 62.7 208167  7.4 239.9  0.3

rz-3x1   16384 16324  4.0 13494  1.6 12118  1.8 147904 43.9 133751  5.6 289.7  0.4
rz-4x1   16384 17522  4.3 13900  1.7 13636  2.1 189491 55.6 194048  9.1 316.7  0.5
rz-5x1   16384 32160  7.9 26957  3.2 24104  3.7 192477 56.3 157096  6.9 328.1  0.6
rz-6x1   16384 35097  8.6 30312  3.6 24239  3.7 172308 50.7 144609  6.7 331.3  0.5
rz-7x1   16384 42069 10.3 35453  4.2 29042  4.4 175760 51.6 180262  7.7 334.5  0.5
rz-8x1   16384 51489 12.6 42490  5.1 32418  4.9 176204 51.8 159905  7.6 337.9  0.6
rz-9x1   16384 64666 16.0 54960  6.6 42246  6.2 191484 56.3 196694  9.3 340.2  0.6
rz-10x1  16384 69135 16.9 59571  7.0 43232  6.3 171677 50.6 180785  8.4 334.8  0.6
rz-10x2  16384 77033 18.7 57714  6.8 46416  6.3 202404 63.2 192582  8.4 383.6  0.6
rz-12x1  16384 89859 22.1 78561  9.1 53677  7.6 182063 52.9 186515  8.3 337.1  0.6
rz-12x4  16384 76116 18.8 66149  7.9 50596  7.0 247209 70.5 227460  9.9 529.4  0.8

rz-10x1  16384 67515 16.6 60519  7.2 42928  6.2 175072 50.7 178989  8.3 337.3  0.6
rz-10x2  16384 69963 17.3 60538  7.4 46981  6.5 169718 49.2 442524 17.6 440.5  0.6
rz-12x1  16384 87009 21.5 78695  9.1 54537  7.8 182325 53.0 186673  8.7 322.3  0.6
rz-12x2  16384 79959 19.7 67420  8.2 49417  7.0 174675 50.7 442809 17.3 452.9  0.6
rz-12x3  16384 61741 15.2 53673  6.6 43119  6.0 206408 59.1 279320 11.9 481.1  0.7
rz-12x4  16384 76765 19.1 66507  8.0 49817  7.0 242528 69.2 230220 10.4 583.1  0.8

rz-10x1  20480 68026 16.7 55075  6.9 42194  6.2 171492 49.8 177911  8.6 211.1  0.5
rz-10x2  20480 69869 17.2 59642  7.3 45595  6.4 190497 55.3 161834  7.6 322.0  0.6
rz-12x1  20480 86353 21.3 76429  9.0 52915  7.6 182706 53.3 187363  8.8 230.5  0.5
rz-12x2  20480 79742 19.6 67190  8.1 48515  6.9 174451 50.6 167175  8.0 296.2  0.5
rz-12x3  20480 61376 15.1 52685  6.9 41649  6.1 177622 51.7 300640 13.0 368.0  0.6
rz-12x4  20480 73108 18.1 66084  8.1 47946  6.7 202205 58.6 286075 12.3 418.5  0.7

r2-4x1   16384 13143  3.2 11001  1.4 10177  1.6 146946 43.5 160208  6.7 328.6  0.5
r2-5x1   16384 16728  4.1 14137  1.7 12561  2.0 191950 57.1 152697  7.3 326.7  0.4
r2-6x1   16384 30136  7.4 25255  3.1 22291  3.5 189985 56.0 179590  8.0 305.9  0.6
r2-7x1   16384 32126  7.9 27515  3.3 23750  3.7 161149 47.7 161066  7.7 339.5  0.5
r2-8x1   16384 42141 10.4 36092  4.3 30090  4.6 159255 47.1 139537  6.7 353.9  0.6
r2-9x1   16384 55639 13.6 42627  5.1 32860  4.8 167834 49.6 165601  7.6 323.4  0.5
r2-10x1  16384 60597 14.9 51777  6.3 42409  6.2 192707 57.1 198280  9.7 335.7  0.6
r2-10x2  16384 36511 12.6 29892  3.6 26526  4.0 212053 61.1 259591 12.0 425.2  0.7
```
To contrast the numbers above I have two other pools on the same server

A raidz of 3 1TB disks (Samsung Spinpoint F3) connected to a cheap PCI card (not PCIe!)

```
-------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
rz-3x1   16384 20627  5.0 21695  2.6 13700  2.0 95670 28.0 181541  7.3 266.9  0.4
```
Finally 4 500Gb Samsung Spinpoint F1 connected to onboard sata-ports

```
-------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
rz-4x1   16384 129757 31.9 119700 13.4 61425  8.3 148144 43.4 136389  6.1 254.6  0.4
```


----------

