# How to properly test SSD performance



## Ben (Nov 20, 2013)

Hi,

I am working on the following filesystems setup and would like to do some benchmarks to figure out if everything works as expected.


2x SSD Samsung 840 EVO 400 GB, ZFS-Mirror, AHCI/onBoard
1x SSD Intel 910 800 GB (= 4x 200 GB), ZFS RAID10, PCI-e

Both drives, especially the Intel 910, should provide very good performance. The system is equipped with 64 GB ECC RAM and a Intel E5-2620v2 CPU.

I tested both filesystems with benchmarks/bonnie++ using the following command:
`bonnie++ -d /tmp/ -s 128g -m ssd -f -b -u root`

If I interpreted the output correctly it said 377 MB/s sequential writing, 115 MB/s sequential read, 258 MB/s random read. I am not sure if this is correct.

If it is correct random read would be fast than squential read? And the performance would not be satisfying.

How can I properly test the speed?

The bonnie++ Output is the following:

```
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
		Concurrency   1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
		Machine        Size	K/sec    %CP K/sec    %CP       K/sec    %CP K/sec %CP K/sec %CP  /sec %CP
		node04         128G           377309  78     115387  26           258251  18     5867   82
		Latency                         267ms     825ms             23339us    6541us
```


----------



## User23 (Nov 20, 2013)

Try benchmarks/iozone.


----------



## Ben (Nov 20, 2013)

Ok I did, this is my "Random Write Report" on the SSD:

```
4	8	16	        32	        64	        128	        256	        512	        1024	        2048	        4096	        8192	        16384
64	426'219	760'859	1'392'258	2'358'717	3'057'153								
128	444'133	836'500	1'504'659	2'558'895	3'380'677	4'267'461							
256	429'530	799'625	1'489'641	2'539'563	3'410'808	4'332'998	4'404'088						
512	420'020	815'353	1'510'391	2'665'014	3'415'178	4'610'256	4'375'425	4'742'616					
1024	432'576	824'400	1'541'732	2'694'787	3'591'693	4'783'849	4'674'510	4'876'180	4'949'226				
2048	423'842	831'567	1'460'921	2'684'517	3'730'203	4'751'580	4'728'042	4'829'046	4'934'459	5'095'447			
4096	434'583	803'443	1'553'220	2'673'574	3'743'865	4'801'789	4'768'469	4'922'872	5'032'468	5'113'352	5'131'680		
8192	424'940	828'728	1'538'118	2'733'296	3'813'593	5'000'831	4'673'646	4'937'591	5'088'216	5'136'133	5'165'475	4'538'452	
16384	421'713	787'346	1'448'866	2'479'074	3'704'393	4'831'639	4'624'209	4'741'954	4'721'753	4'662'486	4'504'775	4'158'564	3'457'269
32768	0	0	0	        0	        3'369'113	4'298'620	4'202'665	4'363'306	4'417'440	4'478'909	4'419'144	4'032'974	3'800'140
65536	0	0	0	        0	        3'229'344	4'265'256	4'162'621	4'300'490	4'452'126	4'339'596	4'463'332	4'012'768	3'839'516
131072	0	0	0	        0	        3'373'898	4'253'238	4'190'697	4'341'440	4'411'393	4'440'863	4'387'208	3'954'400	3'815'554
262144	0	0	0	        0	        3'350'261	3'814'281	4'233'860	4'309'869	4'295'826	4'464'463	4'470'780	3'961'435	3'389'894
524288	0	0	0	        0	        3'166'984	3'699'724	4'068'695	4'343'296	4'375'270	4'039'976	4'451'339	3'922'345	3'662'274
```
This means this constellation reaches out to around 4 GB/s random write?


----------



## User23 (Nov 20, 2013)

Ben said:
			
		

> Ok I did, this is my "Random Write Report" on the SSD:
> 
> This means this constellation reaches out to around 4 GB/s random write?



Hm, 4GB/s can*'*t be, if the specs say 2 / 1 GB/s sequential read / write.

Always show the command line you used for iozone.


----------



## Ben (Nov 20, 2013)

Sorry:
`iozone -a -+u -b output.xls`


----------



## User23 (Nov 21, 2013)

```
OPTIONS
       -a     Used to select full automatic mode. Produces output that  covers
              all  tested  file  operations  for record sizes of 4k to 16M for
              file sizes of 64k to 512M.
```

Test it with at least double the RAM you use. 


```
-s 128g
```


----------



## Ben (Nov 22, 2013)

I asked it to use 128 GB and it is running for 20 hours now. I guess this is normal?

I replaced the Riser-Card and set the PCIe-Port in the BIOS to x8 GEN2.

`diskinfo -t da0` stills says around 200 MB/s. But I will wait for iozone to finish.


----------



## User23 (Nov 22, 2013)

You should read the manual maybe. I never used the automatic mode.
Maybe the test is now running 13 times with different record sizes, for file sizes from 64k up the maximum file size of 128GB, which are 22 tests. 13 * 22 = 286 tests with growing file size.

Mostly *I* use iozone with following commands and change it to the amount of threads, record sizes and file sizes *I* need.

`iozone -R -l 8 -u 8 -r 32k -s 2048m -F /vpool/0 /vpool/1 /vpool/2 /vpool/3 /vpool/4 /vpool/5 /vpool/6 /vpool/7`


----------



## Ben (Nov 22, 2013)

Ok, thanks. I used your settings but reduced to only 1 process.

This gives me the following result for a normal SSD on another system (FreeBSD 9.2, mirrored ZFS pool):

```
Record Size 4 KB
        File size set to 67108864 KB
        Command line used: iozone -R -l 1 -r 4k -s 64g -F /tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 1
        Max process = 1
        Throughput test with 1 process
        Each process writes a 67108864 Kbyte file in 4 Kbyte records

        Children see throughput for  1 initial writers  =  239524.30 KB/sec
        Parent sees throughput for  1 initial writers   =  236847.15 KB/sec
        Min throughput per process                      =  239524.30 KB/sec
        Max throughput per process                      =  239524.30 KB/sec
        Avg throughput per process                      =  239524.30 KB/sec
        Min xfer                                        = 67108864.00 KB

        Children see throughput for  1 rewriters        =  240056.75 KB/sec
        Parent sees throughput for  1 rewriters         =  238077.81 KB/sec
        Min throughput per process                      =  240056.75 KB/sec
        Max throughput per process                      =  240056.75 KB/sec
        Avg throughput per process                      =  240056.75 KB/sec
        Min xfer                                        = 67108864.00 KB
```

For the Intel 910 I get:

```
Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 1
        Max process = 1
        Throughput test with 1 process
        Each process writes a 134217728 Kbyte file in 4 Kbyte records

        Children see throughput for  1 initial writers  =  470988.72 KB/sec
        Parent sees throughput for  1 initial writers   =  467821.71 KB/sec
        Min throughput per process                      =  470988.72 KB/sec
        Max throughput per process                      =  470988.72 KB/sec
        Avg throughput per process                      =  470988.72 KB/sec
        Min xfer                                        = 134217728.00 KB

        Children see throughput for  1 rewriters        =  119654.48 KB/sec
        Parent sees throughput for  1 rewriters         =  119588.09 KB/sec
        Min throughput per process                      =  119654.48 KB/sec
        Max throughput per process                      =  119654.48 KB/sec
        Avg throughput per process                      =  119654.48 KB/sec
        Min xfer                                        = 134217728.00 KB
```


----------



## Ben (Nov 22, 2013)

SSD:

```
Children see throughput for  1 readers          = 1107272.62 KB/sec
        Parent sees throughput for  1 readers           = 1107214.46 KB/sec
        Min throughput per process                      = 1107272.62 KB/sec
        Max throughput per process                      = 1107272.62 KB/sec
        Avg throughput per process                      = 1107272.62 KB/sec
        Min xfer                                        = 67108864.00 KB
```

Intel:

```
Children see throughput for  1 readers          =  192756.50 KB/sec
        Parent sees throughput for  1 readers           =  192756.38 KB/sec
        Min throughput per process                      =  192756.50 KB/sec
        Max throughput per process                      =  192756.50 KB/sec
        Avg throughput per process                      =  192756.50 KB/sec
        Min xfer                                        = 134217728.00 KB
```

This tells me that the SSD writes at *239 MB/s*.
The Intel 910 writes at *470 MB/s*.

The SSD rewrites at *240MB/s*, the Intel at *119 MB/s*.
The SSD reads at *1100 MB/s*, the Intel at *192 MB/s*.

The SSDs are a mirrored zpool, the Intel is a RAID10.


----------



## Ben (Nov 22, 2013)

What could be wrong here? Is it the RAID10? Is it the driver?

All I did was setting:

```
vfs.zfs.prefetch_disable="1"
vm.kmem_size="5G"
```


----------



## kpa (Nov 22, 2013)

Ben said:
			
		

> What could be wrong here? Is it the RAID10? Is it the driver?
> 
> All I did was setting:
> 
> ...



Lose the second vm.kmem_size setting, you shouldn't be touching it unless you really know what it does.


----------



## Ben (Nov 22, 2013)

I removed both settings an am running the tests again. If there is a difference I will just add the first one again.


----------



## Ben (Nov 22, 2013)

Both servers have the same settings now, but the Intel drops the write speed to* 120 MB/s*.


```
Children see throughput for  1 initial writers  =  120045.17 KB/sec
Parent sees throughput for  1 initial writers   =  120030.12 KB/sec
Min throughput per process                      =  120045.17 KB/sec
Max throughput per process                      =  120045.17 KB/sec
Avg throughput per process                      =  120045.17 KB/sec
Min xfer                                        = 134217728.00 KB
```


----------



## kpa (Nov 22, 2013)

Are the partitions on the SSDs aligned in any way? The SSDs may need partitions aligned on rather large blocksize, even as large as 512KB (1024 blocks).


----------



## User23 (Nov 22, 2013)

Test with at least 2x physical RAM. That is why the SSD read is shown faster as its Interface allow it to be 

If you test with 4k record size, the possible amount of IO ops could be reached, *I* don*'*t know.

119 MB/s with 4k record size = 121.856 I/O ops, *I* don*'*t know exactly how this is handled.

I would try it with bigger file size and bigger record size.


```
Test 8x 16GB on 32GB RAM Server

ada2: <Samsung SSD 840 Series DXT06B0Q> ATA-9 SATA 3.x device
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad20


iozone -R -l 8 -u 8 -r 128k -s 16384m -+u -F /mnt/0 /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5 /mnt/6 /mnt/7
       Iozone: Performance Test of File I/O
               Version $Revision: 3.397 $
               Compiled for 64 bit mode.
               Build: freebsd 

       Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                    Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                    Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                    Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                    Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                    Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                    Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                    Ben England.

       Run began: Thu Apr  4 13:32:49 2013

       Excel chart generation enabled
       Record Size 128 KB
       File size set to 16777216 KB
       CPU utilization Resolution = 0.000 seconds.
       CPU utilization Excel chart enabled
       Command line used: iozone -R -l 8 -u 8 -r 128k -s 16384m -+u -F /mnt/0 /mnt/1 /mnt/2 /mnt/3 /mnt/4 /mnt/5 /mnt/6 /mnt/7
       Output is in Kbytes/sec
       Time Resolution = 0.000001 seconds.
       Processor cache size set to 1024 Kbytes.
       Processor cache line size set to 32 bytes.
       File stride size set to 17 * record size.
       Min process = 8 
       Max process = 8 
       Throughput test with 8 processes
       Each process writes a 16777216 Kbyte file in 128 Kbyte records

       Children see throughput for  8 initial writers  =  341829.67 KB/sec
       Parent sees throughput for  8 initial writers   =  332941.41 KB/sec
       Min throughput per process                      =   41777.26 KB/sec 
       Max throughput per process                      =   43736.70 KB/sec
       Avg throughput per process                      =   42728.71 KB/sec
       Min xfer                                        = 16025856.00 KB
       CPU Utilization: Wall time  393.616    CPU time  550.436    CPU utilization 139.84 %


       Children see throughput for  8 rewriters        =  322777.81 KB/sec
       Parent sees throughput for  8 rewriters         =  322729.37 KB/sec
       Min throughput per process                      =   37427.73 KB/sec 
       Max throughput per process                      =   43884.37 KB/sec
       Avg throughput per process                      =   40347.23 KB/sec
       Min xfer                                        = 14308864.00 KB
       CPU utilization: Wall time  382.336    CPU time  213.565    CPU utilization  55.86 %


       Children see throughput for  8 readers          =  512608.18 KB/sec
       Parent sees throughput for  8 readers           =  512546.56 KB/sec
       Min throughput per process                      =   57375.52 KB/sec 
       Max throughput per process                      =   68847.92 KB/sec
       Avg throughput per process                      =   64076.02 KB/sec
       Min xfer                                        = 13981568.00 KB
       CPU utilization: Wall time  243.708    CPU time  142.324    CPU utilization  58.40 %


       Children see throughput for 8 re-readers        =  512178.48 KB/sec
       Parent sees throughput for 8 re-readers         =  512133.55 KB/sec
       Min throughput per process                      =   60422.52 KB/sec 
       Max throughput per process                      =   66913.14 KB/sec
       Avg throughput per process                      =   64022.31 KB/sec
       Min xfer                                        = 15150080.00 KB
       CPU utilization: Wall time  250.751    CPU time  153.099    CPU utilization  61.06 %


       Children see throughput for 8 reverse readers   =  601956.09 KB/sec
       Parent sees throughput for 8 reverse readers    =  601941.13 KB/sec
       Min throughput per process                      =   63409.06 KB/sec 
       Max throughput per process                      =   83636.63 KB/sec
       Avg throughput per process                      =   75244.51 KB/sec
       Min xfer                                        = 12719872.00 KB
       CPU utilization: Wall time  200.601    CPU time   98.322    CPU utilization  49.01 %


       Children see throughput for 8 stride readers    =  520877.82 KB/sec
       Parent sees throughput for 8 stride readers     =  520738.82 KB/sec
       Min throughput per process                      =   64537.12 KB/sec 
       Max throughput per process                      =   65805.98 KB/sec
       Avg throughput per process                      =   65109.73 KB/sec
       Min xfer                                        = 16453760.00 KB
       CPU utilization: Wall time  254.953    CPU time  113.887    CPU utilization  44.67 %


       Children see throughput for 8 random readers    =  524543.39 KB/sec
       Parent sees throughput for 8 random readers     =  524534.18 KB/sec
       Min throughput per process                      =   64954.17 KB/sec 
       Max throughput per process                      =   66153.70 KB/sec
       Avg throughput per process                      =   65567.92 KB/sec
       Min xfer                                        = 16472960.00 KB
       CPU utilization: Wall time  253.611    CPU time  115.681    CPU utilization  45.61 %


       Children see throughput for 8 mixed workload    =  278617.52 KB/sec
       Parent sees throughput for 8 mixed workload     =  118726.60 KB/sec
       Min throughput per process                      =   19902.38 KB/sec 
       Max throughput per process                      =   49650.50 KB/sec
       Avg throughput per process                      =   34827.19 KB/sec
       Min xfer                                        = 6727808.00 KB
       CPU utilization: Wall time  787.832    CPU time 1224.430    CPU utilization 155.42 %


       Children see throughput for 8 random writers    =   98646.83 KB/sec
       Parent sees throughput for 8 random writers     =   97500.60 KB/sec
       Min throughput per process                      =   12166.53 KB/sec 
       Max throughput per process                      =   12476.86 KB/sec
       Avg throughput per process                      =   12330.85 KB/sec
       Min xfer                                        = 16359936.00 KB
       CPU utilization: Wall time 1354.977    CPU time 3285.416    CPU utilization 242.47 %


       Children see throughput for 8 pwrite writers    =  342540.44 KB/sec
       Parent sees throughput for 8 pwrite writers     =  332509.02 KB/sec
       Min throughput per process                      =   41696.94 KB/sec 
       Max throughput per process                      =   43907.32 KB/sec
       Avg throughput per process                      =   42817.56 KB/sec
       Min xfer                                        = 15932672.00 KB
       CPU utilization: Wall time  393.036    CPU time  526.982    CPU utilization 134.08 %


       Children see throughput for 8 pread readers     =  514918.66 KB/sec
       Parent sees throughput for 8 pread readers      =  514869.55 KB/sec
       Min throughput per process                      =   59256.34 KB/sec 
       Max throughput per process                      =   71075.32 KB/sec
       Avg throughput per process                      =   64364.83 KB/sec
       Min xfer                                        = 13987584.00 KB
       CPU utilization: Wall time  236.074    CPU time  132.592    CPU utilization  56.17 %
```


----------



## Ben (Nov 22, 2013)

The second server has 32 GB of RAM, I used a 64 GB file. The Intel 910 host has 64 GB of RAM, here I used the 128 GB file as you suggested before.

I will try again with 16k record size.

Thanks for your help!

EDIT: If I use 8 threads at the same time it will create 8x 128 GB files, right? That will be too much for the SSD


----------



## User23 (Nov 22, 2013)

8x 16 should be enough ^^


----------



## Ben (Nov 22, 2013)

The test is still running:

`iozone -R -l 8 -u 8 -r 128k -s 16384m -+u -F /data/0 /data/1 /data/2 /data/3 /data/4 /data/5 /data/6 /data/7`

The first results (initial writers) say:

```
Children see throughput for  8 initial writers  =  497817.21 KB/sec
Parent sees throughput for  8 initial writers   =  481353.21 KB/sec
Min throughput per process                      =   60812.71 KB/sec 
Max throughput per process                      =   63396.01 KB/sec
Avg throughput per process                      =   62227.15 KB/sec
Min xfer                                        = 16093568.00 KB
CPU Utilization: Wall time  270.807    CPU time   58.883    CPU utilization  21.74 %
```
First I will convert the RAID10 to RAID1, if no difference I will install a *urgh* CentOS to test.

If anybody has an idea what I am doing wrong...

Here is the complete result: http://pastebin.com/iX0trLF6
What do you think? To me it looks strange.

EDIT: SSD ZFS RAID1 on another server: http://pastebin.com/NVLAbFEp

EDIT: SSD ZFS RAID1 on the same server like the Intel: http://pastebin.com/nFJHFhvn

To me it looks like the "Intel"-Server has a general performance problem compared to the other SSD-system. Don't you think? The other one is also FreeBSD 9.2 but another CPU and 32 GB RAM.


----------



## Jon Strabala (Feb 12, 2018)

Hi Ben,

I think you might have some form of ZFS compression enabled and that can drastically skew your results (seems like iozone output is highly compressible).  _My typical datasets compress by at least 2X, not sure how compressible the iozone data is but I am guessing 10X._

For example I have a mirrored pair of 1TB NVMe sticks (I run SmartOS on a 12 core system) for my ZFS test(s) below and the system runs iozone 11X faster when is on e.g. compression=lz4 as compared is off, e.g. compression=none

_Note I didn't bother disabling my ZFS ARC, I was only interested in why my system was almost hitting 10GB/sec. on some iozone write tests._

[FONT=courier new]Children see throughput for  8 initial writers  = 11702057.00 kB/sec *for compression=lz4*
Children see throughput for  8 initial writers  =  1019403.91 kB/sec *for compression=none*[/FONT]

The below chart shows the same test run on the same system with and without compression.






*with lz4 compression* 


```
Run began: Mon Feb 12 01:35:00 2018

        Excel chart generation enabled
        Record Size 128 kB
        File size set to 16777216 kB
        CPU utilization Resolution = 0.000 seconds.
        CPU utilization Excel chart enabled
        Command line used: /tmp/iozone -R -l 8 -u 8 -r 128k -s 16384m -+u -F /zones/temp/0 /zones/temp/1 /zones/temp/2 /zones/temp/3 /zones/temp/4 /zones/temp/5 /zones/temp/6 /zones/temp/7
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 8
        Max process = 8
        Throughput test with 8 processes
        Each process writes a 16777216 kByte file in 128 kByte records

        Children see throughput for  8 initial writers  = 11702057.00 kB/sec
        Parent sees throughput for  8 initial writers   = 10979251.95 kB/sec
        Min throughput per process                      = 1422315.88 kB/sec
        Max throughput per process                      = 1529217.75 kB/sec
        Avg throughput per process                      = 1462757.12 kB/sec
        Min xfer                                        = 15604224.00 kB
        CPU Utilization: Wall time   11.577    CPU time   54.617    CPU utilization 471.77 %


        Children see throughput for  8 rewriters        = 10547577.75 kB/sec
        Parent sees throughput for  8 rewriters         = 10377195.51 kB/sec
        Min throughput per process                      = 1276380.50 kB/sec
        Max throughput per process                      = 1357803.38 kB/sec
        Avg throughput per process                      = 1318447.22 kB/sec
        Min xfer                                        = 15771136.00 kB
        CPU utilization: Wall time   12.357    CPU time   53.642    CPU utilization 434.12 %

        Children see throughput for  8 readers          = 14078536.50 kB/sec
        Parent sees throughput for  8 readers           = 14075299.42 kB/sec
        Min throughput per process                      = 1741966.12 kB/sec
        Max throughput per process                      = 1770798.75 kB/sec
        Avg throughput per process                      = 1759817.06 kB/sec
        Min xfer                                        = 16504960.00 kB
        CPU utilization: Wall time    9.475    CPU time   75.791    CPU utilization 799.91 %
           *
           *
```
*with no compression *


```
Run began: Mon Feb 12 01:54:11 2018

        Excel chart generation enabled
        Record Size 128 kB
        File size set to 16777216 kB
        CPU utilization Resolution = -0.000 seconds.
        CPU utilization Excel chart enabled
        Command line used: /tmp/iozone -R -l 8 -u 8 -r 128k -s 16384m -+u -F /zones/nocomp/0 /zones/nocomp/1 /zones/nocomp/2 /zones/nocomp/3 /zones/nocomp/4 /zones/nocomp/5 /zones/nocomp/6 /zones/nocomp/7
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 8
        Max process = 8
        Throughput test with 8 processes
        Each process writes a 16777216 kByte file in 128 kByte records

        Children see throughput for  8 initial writers  = 1019403.91 kB/sec
        Parent sees throughput for  8 initial writers   =  993211.05 kB/sec
        Min throughput per process                      =  126838.98 kB/sec
        Max throughput per process                      =  127952.06 kB/sec
        Avg throughput per process                      =  127425.49 kB/sec
        Min xfer                                        = 16631296.00 kB
        CPU Utilization: Wall time  131.781    CPU time  207.768    CPU utilization 157.66 %


        Children see throughput for  8 rewriters        =  998865.34 kB/sec
        Parent sees throughput for  8 rewriters         =  957616.15 kB/sec
        Min throughput per process                      =  124569.84 kB/sec
        Max throughput per process                      =  125157.23 kB/sec
        Avg throughput per process                      =  124858.17 kB/sec
        Min xfer                                        = 16698624.00 kB
        CPU utilization: Wall time  134.051    CPU time  172.092    CPU utilization 128.38 %


        Children see throughput for  8 readers          = 2208100.62 kB/sec
        Parent sees throughput for  8 readers           = 2207987.63 kB/sec
        Min throughput per process                      =  275627.53 kB/sec
        Max throughput per process                      =  276785.69 kB/sec
        Avg throughput per process                      =  276012.58 kB/sec
        Min xfer                                        = 16707072.00 kB
        CPU utilization: Wall time   60.615    CPU time   47.610    CPU utilization  78.55 %
           *
           *
```


----------



## ralphbsz (Feb 12, 2018)

A: What is your goal?  What is your workload?  What are the real performance needs of your workload?  Why do you care how fast your SSD is?

My answer would be: You have some application or workload in mind, you have a desired performance in mind (sort of a SLA), you know the "speeds and feeds" (how much IO the workloads wants to do, and how fast your storage system runs), and you are either satisfied or not.

B: Your tests are mostly testing file system effects, such as caching.  If you are trying to determine the raw performance of the SSD itself (without testing the file system), that is a bad thing.  However, getting back to question A above, what is the point of testing SSD performance?  If you are trying to determine the end-to-end performance of your application, then testing with file system effects is exactly the right thing to do, but simple benchmarks like bonnie or iozone won't give you realistic answers.

The best benchmark is your workload itself.  Use the system as intended, and measure how long real-world operations take.  Decide whether this is good enough or not.  If it is not, then and only then is it time to look underneath the covers, and find out where the bottlenecks are.  To do that, you need to look at individual components of your stack, such as the file system, the compression in the file system (if any), the RAID layer (in hardware, software, or none), the OS'es block layer, the communication channels (SATA, SAS, ...), and the end device itself.  

If you have really determined that the SSD itself is the bottleneck (evidence, please!), then the right way to measure and optimize its performance is not to go through the file system, but use specialized test software that talks directly to the SSD block device.  This will give you a baseline performance for the device itself, and then you can see whether other bottlenecks in the system spoil that.  But please be warned: write testing on SSDs will wear them out, and rapid testing (at ~1/2 GByte/s) can reach the wearout limit of SSDs pretty rapidly.  Modern consumer SSDs already have pretty limited write endurance, and strenuous testing may take years off their life.


----------



## Jon Strabala (Feb 12, 2018)

*ralphbsz: *I assume you are responding to me (e.g. ours are recent posts).  Please understand I was pointing out that using "_iozone_" on a filesystem that has any sort of compression is a nonstarter for benchmarking (this was meant for Ben).  I merely gave examples of the issue that I thought Ben was experiencing, e.g. file system effects which almost eliminate I/O when compression is enabled due to the compressible nature of "_iozone_" writes.


----------



## ralphbsz (Feb 12, 2018)

Actually, I was responding to Ben, the original poster.

And I completely agree with you: Using iozone, bonnie and friends to test *SSD* performance and going through a file system *with compression* is meaningless.  Testing device performance through a file system is already somewhat silly: it is really hard to get rid of file system effects, most importantly caching, but also the conversion between random and sequential depending on FS layout.  But if one adds compression, then the test data has to compress about as well as the real workload data, which in the case of iozone/bonnie/... is guaranteed to not be the case (and wrong by a large factor).

But going even further: In most cases, I don't see the point of testing raw device performance; what the end user really wants is application performance, and the bottlenecks are commonly not identical to the performance of the storage device.


----------



## SirDice (Feb 14, 2018)

Note that this thread actually started more than 4 years ago and Ben hasn't been seen here since February last year.


----------

