# LSI 1068e, Rackable SE3016 200MB/sec Ceiling?



## ZFSZealot (Oct 24, 2013)

I am using: FreeBSD 9.1-RELEASE-p7 amd64.

System hardware:

Supermicro X7SBE
Intel Q9300 (Core 2 Quad)
Onboard Intel ICH-9 SATA-II ports
Onboard Intel 82573E/L 1 Gbps Ethernet ports
Onboard ATI ES1000 video
IBM BR10i (crossflashed to LSI SAS3081e, 1068e controller) as PCI-Express 8x card (Fusion-MPT SAS)
Supermicro AOC-SAT2-MV8 (MV88SX6081 8-port SATA II controller) on PCI-X card
Adaptec ASC-39320 SCSI controller on PCI-X card
Silicon Image PCI0680 PCI ATA-133 host controller on PCI card
Dual 82546EB gigabit Ethernet controller on PCI-X card
2x Supermicro CSE-M35S enclosure attached to ASC-39320
3x HTS721010G9SA00 (Hitachi 2.5" 100 GB SATAI) attached to ICH-9
2x SSDSA2CW120G3 (Intel 120 GB SATAII SSD) attached to ICH-9
1x Samsung UM410 (Samsung 16 GB SATAII SSD) attached to ICH-9
8x Fujitsu MBA3300NC, four in each CSE-M35S attached to ASC-39320
8x Hitachi HDS721010CLA332 attached to AOC-SAT2-MV8[/file]
4x BenQ DW1640/DW1650 (DVD+RW drive) attached to PCI0680
1x Rackable (SGI) SE3016 attached to BR10i
Assortment of drives in the SE3016 on the BR10i, including Fujitsu MBA3300RC, Toshiba DT01ACA200, Hitachi HTS721010G9SA00, etc.

Output from `camcontrol devlist`:

```
root@cadence:/tmp # camcontrol devlist
<BENQ DVD DD DW1640 BSLB>          at scbus0 target 0 lun 0 (cd0,pass0)
<BENQ DVD DD DW1640 BSLB>          at scbus0 target 1 lun 0 (cd1,pass1)
<BENQ DVD DD DW1640 BSLB>          at scbus1 target 0 lun 0 (cd2,pass2)
<BENQ DVD DD DW1650 BCIC>          at scbus1 target 1 lun 0 (cd3,pass3)
<FUJITSU MBA3300NC 0102>           at scbus2 target 0 lun 0 (da0,pass4)
<FUJITSU MBA3300NC 0102>           at scbus2 target 1 lun 0 (da1,pass5)
<FUJITSU MBA3300NC 4801>           at scbus2 target 2 lun 0 (da2,pass6)
<FUJITSU MBA3300NC 0102>           at scbus2 target 4 lun 0 (da3,pass7)
<SUPER GEM318 0>                   at scbus2 target 6 lun 0 (ses0,pass8)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 0 lun 0 (da4,pass9)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 1 lun 0 (da5,pass10)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 2 lun 0 (da6,pass11)
<FUJITSU MBA3300NC 4801>           at scbus3 target 4 lun 0 (da7,pass12)
<SUPER GEM318 0>                   at scbus3 target 6 lun 0 (ses1,pass13)
<Hitachi HDS721010CLA332 JP4OA25C>  at scbus4 target 0 lun 0 (ada0,pass14)
<Hitachi HDS721010CLA332 JP4OA25C>  at scbus5 target 0 lun 0 (ada1,pass15)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus6 target 0 lun 0 (ada2,pass16)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus7 target 0 lun 0 (ada3,pass17)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus8 target 0 lun 0 (ada4,pass18)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus9 target 0 lun 0 (ada5,pass19)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus10 target 0 lun 0 (ada6,pass20)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus11 target 0 lun 0 (ada7,pass21)
<ATA TOSHIBA DT01ACA2 ABB0>        at scbus12 target 7 lun 0 (pass40,da19)
<ATA ST3250824AS H>                at scbus12 target 8 lun 0 (pass34,da13)
<RACKABLE SE3016-SAS 0227>         at scbus12 target 9 lun 0 (pass29,ses2)
<ATA ST3160815AS A>                at scbus12 target 10 lun 0 (pass38,da17)
<ATA ST3250824AS H>                at scbus12 target 12 lun 0 (pass39,da18)
<ATA Hitachi HDS72101 A3MA>        at scbus12 target 13 lun 0 (pass32,da11)
<ATA Hitachi HDS72101 A3MA>        at scbus12 target 14 lun 0 (pass33,da12)
<ATA ST3250824AS H>                at scbus12 target 15 lun 0 (pass35,da14)
<ATA ST3160815AS A>                at scbus12 target 16 lun 0 (pass37,da16)
<ATA ST3160812AS J>                at scbus12 target 17 lun 0 (pass30,da9)
<ATA ST3250824AS H>                at scbus12 target 18 lun 0 (pass36,da15)
<ATA Hitachi HDS72101 A39C>        at scbus12 target 19 lun 0 (pass31,da10)
<FUJITSU MBA3300RC D305>           at scbus12 target 20 lun 0 (pass28,da8)
<Hitachi HTS721010G9SA00 MCZOC10H>  at scbus13 target 0 lun 0 (ada8,pass22)
<Hitachi HTS721010G9SA00 MCZOC10H>  at scbus14 target 0 lun 0 (ada9,pass23)
<HTS721010G9SA00 MCZOC10V>         at scbus15 target 0 lun 0 (ada10,pass24)
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q>  at scbus16 target 0 lun 0 (ada11,pass25)
<INTEL SSDSA2CW120G3 4PC10362>     at scbus17 target 0 lun 0 (ada12,pass26)
<INTEL SSDSA2CW120G3 4PC10362>     at scbus18 target 0 lun 0 (ada13,pass27)
```

I'm seeing very stable but slow performance from drives in the Rackable SE3016 on the BR10i/LSI SAS3081e.  Using dd to do large sequential reads from one drive does not saturate the bandwidth, but two or more seem to hit a limit at about 200 MB/sec.  I'm using `gstat` to determine this.  The transfer rates on drives on scbus12 never add up to more than about 200 MB/sec.  I'm using, for example:


```
dd bs=256k if=/dev/da8 of=/dev/null
dd bs=256k if=/dev/da9 of=/dev/null
dd bs=256k if=/dev/da10 of=/dev/null
dd bs=256k if=/dev/da11 of=/dev/null
```

In the example given, I see about 50 MB/sec from all four drives.  I see very similar behavior after constructing a ZFS pool and scrubbing.  Scrubs of terabytes of data do not show any errors.

The Rackable enclosure provides expander functionality but the OEM expander manufacturer is undetermined (camcontrol identifies as RACKABLE SE3016-SAS 0227).  I'm attaching it with a SFF-8087 to SFF-8088 cable and have tried two different cables with the same result.

I confirmed four PHY links appear to be up and active on the onboard LEDs, so I think I've ruled out my initial suspicion that it wasn't linking up a wide (4 PHY) port.

I've verified that an Adaptec 1045 card on a similar X7SBE running a Linux live CD gets me a much higher ceiling, closer to 800-900 MB/sec, when running similar dd testing.

I attempted to use `camcontrol smpphylist /dev/da8` to examine the SAS topology but I get:


```
camcontrol: error sending command
Segmentation fault (core dumped)
```

Internet searches don't seem to indicate that this is a common problem.  I am planning on booting the main FreeBSD server using a Linux live CD to try to isolate whether this is a hardware or FreeBSD/mpt driver problem but that requires a maintenance window.  Are there any other tools that I can use to further determine the cause?


----------



## ZFSZealot (Nov 9, 2013)

For what it's worth, I bought an IBM M1115 which is basically OEM LSI 9211-8i with the more modern LSI SAS2008 chip.  I crossflashed to the LSI 9211-IT firmware per the procedure at http://www.servethehome.com/ibm-serveraid-m1015-part-4/.  Replacing the 1068e based card with this solved the problem.

I did not have the time or patience to get Linux to boot, long story short booting from a DVD is an issue because of the hardware and I didn't fool around with other methods like making a USB key.

Testing was done with `dd bs=256k if=/dev/daX of=/dev/null &`.

A shot from `gstat` all disks in the SE3016 reading, about 850MB/s, respectable for a Core2 era server:


```
dT: 1.002s  w: 1.000s  filter: ^da[0-9]*$|^ada[0-9]*$
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ada4
    0      0      0      0    0.0      0      0    0.0    0.0| ada5
    0      0      0      0    0.0      0      0    0.0    0.0| ada6
    0      0      0      0    0.0      0      0    0.0    0.0| ada7
    0      0      0      0    0.0      0      0    0.0    0.0| ada8
    0      0      0      0    0.0      0      0    0.0    0.0| ada9
    0      0      0      0    0.0      0      0    0.0    0.0| ada10
    0      0      0      0    0.0      0      0    0.0    0.0| ada11
    0      0      0      0    0.0      0      0    0.0    0.0| ada12
    0      0      0      0    0.0      0      0    0.0    0.0| ada13
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| da5
    0      0      0      0    0.0      0      0    0.0    0.0| da6
    1    552    552  70643    1.8      0      0    0.0   98.3| da7
    1    643    643  82267    1.5      0      0    0.0   98.3| da8
    1    485    485  62084    2.0      0      0    0.0   98.5| da9
    1    496    496  63489    2.0      0      0    0.0   98.4| da10
    1    498    498  63744    2.0      0      0    0.0   98.5| da11
    1    494    494  63234    2.0      0      0    0.0   98.5| da12
    1    433    433  55441    2.3      0      0    0.0   98.7| da13
    1    435    435  55697    2.3      0      0    0.0   98.5| da14
    1    443    443  56719    2.2      0      0    0.0   98.4| da15
    1    638    638  81629    1.5      0      0    0.0   98.3| da16
    1    546    546  69876    1.8      0      0    0.0   98.5| da17
    1    547    547  70004    1.8      0      0    0.0   98.3| da18
    1    556    556  71154    1.8      0      0    0.0   98.6| da19
```

Another showing a few of the newer, better disks at their full sequential transfer rate:


```
dT: 1.002s  w: 1.000s  filter: ^da[0-9]*$|^ada[0-9]*$
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ada4
    0      0      0      0    0.0      0      0    0.0    0.0| ada5
    0      0      0      0    0.0      0      0    0.0    0.0| ada6
    0      0      0      0    0.0      0      0    0.0    0.0| ada7
    0      0      0      0    0.0      0      0    0.0    0.0| ada8
    0      0      0      0    0.0      0      0    0.0    0.0| ada9
    0      0      0      0    0.0      0      0    0.0    0.0| ada10
    0      0      0      0    0.0      0      0    0.0    0.0| ada11
    0      0      0      0    0.0      0      0    0.0    0.0| ada12
    0      0      0      0    0.0      0      0    0.0    0.0| ada13
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| da5
    0      0      0      0    0.0      0      0    0.0    0.0| da6
    1   1445   1445 184974    0.7      0      0    0.0   96.6| da7
    1    963    963 123273    1.0      0      0    0.0   97.7| da8
    0      0      0      0    0.0      0      0    0.0    0.0| da9
    0      0      0      0    0.0      0      0    0.0    0.0| da10
    0      0      0      0    0.0      0      0    0.0    0.0| da11
    0      0      0      0    0.0      0      0    0.0    0.0| da12
    0      0      0      0    0.0      0      0    0.0    0.0| da13
    0      0      0      0    0.0      0      0    0.0    0.0| da14
    0      0      0      0    0.0      0      0    0.0    0.0| da15
    1    944    944 120846    1.0      0      0    0.0   97.7| da16
    0      0      0      0    0.0      0      0    0.0    0.0| da17
    1   1008   1008 129022    1.0      0      0    0.0   97.6| da18
    1   1019   1019 130427    1.0      0      0    0.0   97.5| da19
```


More current `camcontrol devlist`:


```
<BENQ DVD DD DW1640 BSLB>          at scbus0 target 0 lun 0 (cd0,pass0)
<BENQ DVD DD DW1640 BSLB>          at scbus0 target 1 lun 0 (cd1,pass1)
<BENQ DVD DD DW1640 BSLB>          at scbus1 target 0 lun 0 (cd2,pass2)
<BENQ DVD DD DW1650 BCIC>          at scbus1 target 1 lun 0 (cd3,pass3)
<FUJITSU MBA3300NC 0102>           at scbus2 target 0 lun 0 (da0,pass4)
<FUJITSU MBA3300NC 0102>           at scbus2 target 1 lun 0 (da1,pass5)
<FUJITSU MBA3300NC 4801>           at scbus2 target 2 lun 0 (da2,pass6)
<FUJITSU MBA3300NC 0102>           at scbus2 target 4 lun 0 (da3,pass7)
<SUPER GEM318 0>                   at scbus2 target 6 lun 0 (ses0,pass8)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 0 lun 0 (da4,pass9)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 1 lun 0 (da5,pass10)
<FUJITSU MBA3300NC 5E03>           at scbus3 target 2 lun 0 (da6,pass11)
<SUPER GEM318 0>                   at scbus3 target 6 lun 0 (ses1,pass12)
<Hitachi HDS721010CLA332 JP4OA25C>  at scbus4 target 0 lun 0 (ada0,pass13)
<Hitachi HDS721010CLA332 JP4OA25C>  at scbus5 target 0 lun 0 (ada1,pass14)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus6 target 0 lun 0 (ada2,pass15)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus7 target 0 lun 0 (ada3,pass16)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus8 target 0 lun 0 (ada4,pass17)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus9 target 0 lun 0 (ada5,pass18)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus10 target 0 lun 0 (ada6,pass19)
<Hitachi HDS721010CLA332 JP4OA39C>  at scbus11 target 0 lun 0 (ada7,pass20)
<ATA TOSHIBA DT01ACA2 ABB0>        at scbus12 target 8 lun 0 (da7,pass21)
<FUJITSU MBA3300RC D306>           at scbus12 target 9 lun 0 (da8,pass22)
<ATA ST3250824AS H>                at scbus12 target 10 lun 0 (da9,pass23)
<ATA ST3250824AS H>                at scbus12 target 11 lun 0 (da10,pass24)
<ATA ST3250824AS H>                at scbus12 target 12 lun 0 (da11,pass25)
<ATA ST3250824AS H>                at scbus12 target 13 lun 0 (da12,pass26)
<ATA ST3160815AS A>                at scbus12 target 14 lun 0 (da13,pass27)
<ATA ST3160815AS A>                at scbus12 target 15 lun 0 (da14,pass28)
<ATA ST3160812AS J>                at scbus12 target 16 lun 0 (da15,pass29)
<FUJITSU MBA3300RC D305>           at scbus12 target 17 lun 0 (da16,pass30)
<ATA Hitachi HDS72101 A3MA>        at scbus12 target 18 lun 0 (da17,pass31)
<ATA Hitachi HDS72101 A39C>        at scbus12 target 19 lun 0 (da18,pass32)
<ATA Hitachi HDS72101 A3MA>        at scbus12 target 20 lun 0 (da19,pass33)
<RACKABLE SE3016-SAS 0227>         at scbus12 target 21 lun 0 (ses2,pass34)
<Hitachi HTS721010G9SA00 MCZOC10H>  at scbus13 target 0 lun 0 (ada8,pass35)
<Hitachi HTS721010G9SA00 MCZOC10H>  at scbus14 target 0 lun 0 (ada9,pass36)
<HTS721010G9SA00 MCZOC10V>         at scbus15 target 0 lun 0 (ada10,pass37)
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q>  at scbus16 target 0 lun 0 (ada11,pass38)
<INTEL SSDSA2CW120G3 4PC10362>     at scbus17 target 0 lun 0 (ada12,pass39)
<INTEL SSDSA2CW120G3 4PC10362>     at scbus18 target 0 lun 0 (ada13,pass40)
```


----------

