# 2 of 3 M1015 controllers detect HDD's



## thethirdnut (Nov 9, 2012)

Hi All,

Stats:

[Mobo] Supermicro X9SCM-F bios 2.0a - 2 x PCIe-8x +  2 x PCIe-4x [8x slot]
[SAS cards] 3 x IBM M1015 SAS cards; all with IT fw v14; uses LSI SAS2008 chipset
[OS] FreeBSD 9.0-RELEASE
[other] Xeon E1220V2, 16 GB ECC, SSD for OS, 10 x HDD in a Norco RPC-4224

Problem:

- 1 of the 3 verified working IBM M1015 SAS cards does not detect drives behind it.
- this particular card works / detects drives when swapped slots, backplane and when booted off Linux LiveCD.
- problem stays with keeping one of these PCIe 8x cards in a PCIe 4x [electrical, 8x mechanical] slot.
- pciconf detects all 3 x M1015 cards

pciconf http://pastebin.ca/2249885

sysctl http://pastebin.ca/2249886

I am using camcontrol devlist to find drives and no matter what I can't get last card in the PCIe-4x slot to detect HDD's.

I don't know where to go from here - any pointers please?

TIA.


----------



## thethirdnut (Nov 12, 2012)

I apparently don't yet have ability to mark this as SOLVED, but it now is. 

Thanks to @george from this post: http://forums.freebsd.org/showthread.php?p=196651


----------



## boris_net (Nov 13, 2012)

Any benchmark on what kind of read and write transfer rate you can achieve on such system?

It looks a lot like what I have selected and about to buy, would be grateful if you would be ok to share some performance results on this.


----------



## thethirdnut (Nov 13, 2012)

@boris_net

Not as of yet - I am using mainly Green drives so not expecting blazing performance, however, since it's basically going to be an overkill NAS 120 MB/s performance is all I need.

Will post later once I get everything done - SSD does boot damn quick though. Currently trying to figure out why my OpenSSH keys from Linuxland fail with the passphrase.


----------



## boris_net (Nov 13, 2012)

Thanks keep us posted, I am interested in seeing if you can achieve more than 120MB/s locally.


----------



## thethirdnut (Nov 14, 2012)

@boris_net

What would you recommend for benchmarking? 

I'm mainly interested in filesystem sustained reads, writes for use as a NAS. Bonnie++ and iozone seem a little overkill since I'm not going to be creating, deleting hundreds of files on a regular basis.

I may default to my dd created random file xfering back and forth...


----------



## boris_net (Nov 17, 2012)

Agreed dd would be a good start


----------



## thethirdnut (Dec 4, 2012)

Sorry for the delay...will post soon.


----------



## thethirdnut (Dec 4, 2012)

Summary:
-- I am using a 10x 2TB raidz2 pool of cheap â€˜greenâ€™ drives.
-- This box will be a glorified NAS; wonâ€™t do much else. Hardware may be overkill, but I am building this to last. 
-- Am currently getting 85+ MB/s read/write over NFSv4 so I am happy with that...could be better I know, but works for me. On to the next project once this has all stabilized and put into service.
-- I am also aware that dd is not a de facto benchmark. I may try bonnie++ later if time permits.



```
[cpu]		Intel Xeon E3-1220-V2
[mobo]		Supermicro X9SCM-F (bios 2.0a)
[ram]		(4x) Crucial CT51272BA1339 [4GB DDR3 Unbuffered ECC]
[ssd]		Crucial M4 64GB (fw 000F)
[sas card]	(3x) IBM M1015 (IT mode v14)
[case]		Norco RPC-4224
[psu]		Antec Truepower New 750W 
[pwr cable]	Antec 77CM Molex Connector With Cable for NeoPower Series
[sas cable]	LSI Multi-Lane Internal SFF-8087 to SFF-8087 SAS Cable 0.6M	
[hdd]		(4x) 2TB Seagate ST2000DL003
		(3x) 2TB WD WD20EARS
		(1x) 2TB WD WD20EARX
		(1x) 2TB Hitachi HDS5C3020ALA632 
                (1x) 2TB Samsung/Seagate ST2000DL004
[os]		FreeBSD 9.0-RELEASE amd64
[NFS]		v4
[ZFS]		v28, dedupe, compression OFF
[firewall]	ipfw
```

NOTES:
** did NOT do any ZFS tuning - there is no /boot/loader.conf file.
** idle power usage for FULL system is approx 120 Watts.
** I also kept the stock 5 x 80mm fan wall in the Norco - yes it is noisy but it keeps all cool and unit in diff room from HT. 

================================================================
**I created a 24GB random file for benchmarking due to I HOPE this helping saturating the 16GB of RAM**

/dev/urandom â†’ SSD write: 86 MB/s 
--not really accurate measure of SSD write, but needed to make file somehow

```
dd bs=1M count=24000 if=/dev/urandom of=/home/user/testdir/rnd24GB.dd
...
25165824000 bytes transferred in 291.965923 secs (86194388 bytes/sec)
```

SSD read: 573 MB/s

```
dd if=/home/user/testdir/rnd24GB.dd of=/dev/null
...
25165824000 bytes transferred in 46.863236 secs (537005680 bytes/sec)
```

HDD raw write: all 120+ MB/s 
-- sent the rnd24GB.dd file to the devices

```
dd bs=1M count=24000 if=/home/user/testdir/rnd24GB.dd of=/dev/da0
...
14997782528 bytes transferred in 122.560944 secs (122369998 bytes/sec)

dd bs=1M count=24000 if=/home/user/testdir/rnd24GB.dd of=/dev/da1
...
25165824000 bytes transferred in 189.845046 secs (132559814 bytes/sec)
```

SSD â†’ ZFS write: 78 MB/s 	
-- looks like SATA bottleneck since /dev/urandom & NFS better results; see below

```
dd if=/home/user/testdir/rnd24GB.dd of=/home/user/ztank0/file1.dd
...
25165824000 bytes transferred in 322.990155 secs (77915143 bytes/sec)
```

ZFS read: 196 MB/s

```
dd if=/home/user/ztank0/file1.dd of=/dev/null
...
25165824000 bytes transferred in 128.271675 secs (196191591 bytes/sec)
```

/dev/urandom â†’ ZFS write: 85 MB/s 

```
dd bs=1M count=24000 if=/dev/urandom of=/home/user/ztank0/file24GBrand.dd
...
25165824000 bytes transferred in 294.469267 secs (85461632 bytes/sec)
dd bs=1M count=24000 if=/dev/urandom of=/home/user/ztank0/file24GBrand2.dd
...
25165824000 bytes transferred in 293.075589 secs (85868032 bytes/sec)
```

NFSv4 ZFS â†’ Linux client disk write: 76 MB/s

```
dd if=/mnt/zfs/file24GBrand.dd of=/home/user/local/tt_temp/24GBxfer1.dd
...
25165824000 bytes (25 GB) copied, 329.331 s, 76.4 MB/s
```

NFSv4 ZFS â†’ Linux client read (stream): 88 MB/s

```
dd if=/mnt/zfs/file24GBrand.dd of=/dev/null
...
25165824000 bytes (25 GB) copied, 285.951 s, 88.0 MB/s
```

NFSv4 Linux client write â†’ ZFS: 86 MB/s

```
dd if=/home/user/local/tt_temp/24GBxfer1.dd of=/mnt/zfs/xferbak24GB.dd
...
25165824000 bytes (25 GB) copied, 291.752 s, 86.3 MB/s
```

================================================================
ZFS info
-- I used the GPT partition per disk + GNOP for each drive as described here: http://forums.freebsd.org/showpost.php?p=175779&postcount=6
-- scrubbing at 700+ M/s too for ableit a very small test dataset


```
#zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
tank0  1.02M  13.6T   329K  /tank0
# zpool status tank0
 pool: tank0
state: ONLINE
scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank0       ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da4p1   ONLINE       0     0     0
	    da6p1   ONLINE       0     0     0
	    da7p1   ONLINE       0     0     0
	    da8p1   ONLINE       0     0     0
	    da9p1   ONLINE       0     0     0
	    da5p1   ONLINE       0     0     0
	    da3p1   ONLINE       0     0     0
	    da2p1   ONLINE       0     0     0
	    da1p1   ONLINE       0     0     0
	    da0p1   ONLINE       0     0     0

errors: No known data errors

#zpool status -v tank0
  pool: tank0
 state: ONLINE
 scan: scrub in progress since Tue Dec  4 21:01:36 2012
    65.0G scanned out of 123G at 723M/s, 0h1m to go
    0 repaired, 52.69% done
config:

	NAME        STATE     READ WRITE CKSUM
	tank0       ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da4p1   ONLINE       0     0     0
	    da6p1   ONLINE       0     0     0
	    da7p1   ONLINE       0     0     0
	    da8p1   ONLINE       0     0     0
	    da9p1   ONLINE       0     0     0
	    da5p1   ONLINE       0     0     0
	    da3p1   ONLINE       0     0     0
	    da2p1   ONLINE       0     0     0
	    da1p1   ONLINE       0     0     0
	    da0p1   ONLINE       0     0     0

errors: No known data errors
```


----------



## thethirdnut (Dec 4, 2012)

Thank you Mods for the edit ability...added Notes to above post.


----------

