# IBM ServeRaid M1015 and FreeBSD



## Migelo (Dec 19, 2012)

Hi!

I'm looking for a way to expand the number of my SATAII ports and have come across this card: IBM ServeRaid M1015.
I'll be using it just to passthrough the HDDs to the system because of ZFS.

Does it really work without problem with the latest FreeBSD release?

Are there any things I need to do, to get it working?

If you know a better and possibly cheaper solution for at least 8xSATAII ports, please let me know.


----------



## olav (Dec 20, 2012)

Yes that controller works great
http://howtounix.info/man/FreeBSD/man4/mps.4


----------



## Sebulon (Dec 20, 2012)

WerenÂ´t you supposed to flash the firmware into Initiator/Target(IT) mode first, or does it "just work"?

/Sebulon


----------



## Migelo (Dec 20, 2012)

Thanks olav!

Can someone confirm if it "just works" or is flashing necessary?


----------



## Sfynx (Dec 21, 2012)

Apparently you can flash the thing to the well-known LSI 9211-8i firmware: http://www.servethehome.com/ibm-serveraid-m1015-part-4/

I know the mps(4) driver does not support IR mode, but I'm not sure whether that means that the entire driver won't load when using IR firmware, or that it will simply not use the extra functionality that it provides.


----------



## olav (Dec 21, 2012)

I belive mfi(4) should work in IR mode, but I've not tested it yet.


----------



## Migelo (Dec 23, 2012)

If I'm lucky, maybe I will get it before 2013. In either case, I'll be reporting my results.


----------



## Migelo (Dec 23, 2012)

And thanks for the help everyone!


----------



## Migelo (Dec 27, 2012)

Another question. Does this card work in IDE or AHCI mode?


----------



## olav (Dec 28, 2012)

it works as a scsi controller


----------



## m6tt (Dec 28, 2012)

You can run it under mfi driver (with original firmware) or under mps driver (without).

If you want to use "hardware" raid1/raid0/raid10, use mfi
If you want to just pass through disks, flash it to "9210-8i" and use the mps driver.


Cross flashing is not for the faint of heart, and sometimes require trying different computers/BIOS versions to successfully complete. Be aware that cross flashing etc is usually 100% unsupported by LSI or anyone else, not recommended for enterprise production.

Under mfi, un raided disks show up as "/dev/mfisyspd*", with about equal performance to da devices from mps. However, as far as I know you can't get info like linkspeed or drive cache status out of the mfi driver.

If you are using ZFS, you should be able to switch between mps and mfi without doing anything to your pool.


----------



## Migelo (Dec 28, 2012)

Thank you, now I just need to get the card so I can start toying around with it.


----------



## thethirdnut (Jan 5, 2013)

Yes - these cards work well...I had issues with 2 x defective HDD's causing instability, but once replaced has been very fast and stable cards to use. Notes on what I am using below:


```
[cpu]		Intel Xeon E3-1220-V2
[mobo]		Supermicro X9SCM-F (bios 2.0a)
[ram]		(4x) Crucial CT51272BA1339 [4GB DDR3 Unbuffered ECC]
[ssd]		Crucial M4 64GB (fw 000F)
[sas card]	(3x) IBM M1015 (IT mode v15)
[mps driver]	manually loading LSI's mpslsi driver v15
[hdd]		(3x) 2TB Seagate ST2000DL003
		(3x) 2TB WD WD20EARS
		(2x) 2TB WD WD20EFRX (RED)
		(1x) 2TB Hitachi HDS5C3020ALA632 
                (1x) 2TB Samsung/Seagate ST2000DL004
[os]		FreeBSD 9.1-RELEASE amd64
[NFS]		v4
[ZFS]		v28, dedupe, compression OFF
```


----------



## Migelo (Jan 5, 2013)

Was that the card's fault, or the HDD's?

The card is on the way from Austria! And just today, I got the SAS->4xSATA cable, it was very expensive, 26â‚¬(with shipping).......


----------



## thethirdnut (Jan 5, 2013)

HDD's were at fault - both were failing smartctl parameters and longtests once looked into. 

This did not make the mps driver for SAS2008 chipset happy, which in turn ZFS got fed-up and booted the HDD's from the array. This wasn't always the 2 faulty HDD's directly which is why it took me awhile to pinpoint the cause with smartctl longtests. 

Once those 2 x HDD's were replaced I haven't seen a mps driver or ZFS issues since - knock on wood - doing a 9TB transfer of data and several scrubs . ;-)


----------



## Migelo (Jan 5, 2013)

Bad drives, making the controller unhappy xD.

Thanks for your feedback! I'm more and more confident that it'll work w/o problems. (after I can solve the "EncoderFFmpeg.cpp:80: error: 'AVIO_FLAG_READ' was not declared in this scope" when compiling XBMC)


----------



## Migelo (Jan 18, 2013)

Ok, I finally got the card. If I just want to passthrough the disks I can just use the MPS drivers by adding 

```
mps_load="YES"
```

??


----------



## Sebulon (Jan 18, 2013)

Since itÂ´s included in GENERIC, itÂ´ll "just work"

/Sebulon


----------



## Migelo (Jan 18, 2013)

Are you sure?

I get 

```
mfi0: command 0xffffff8001b9e44o timeout after 59 seconds
```

And the it just repeats with increasing numbers of seconds.
It happens with or without disks attached.


----------



## thethirdnut (Jan 18, 2013)

What I did - two main steps is update the M1015's firmware + dload and load LSI's version of the firmware.

All files can be found here: http://www.lsi.com/support/Pages/Do...Host Bus Adapters&productname=LSI SAS 9211-8i

*1) Flash the M1015 to the v15 'IT-mode' firmware.*

LSI9240(IBM M1015) to a LSI9211-IT mode


```
1) Boot with FreeDOS USB drive with all dloaded stuff on it.
2) megarec -writesbr 0 sbrempty.bin
3) megarec -cleanflash 0
4) Boot into BIOS, Exit, Boot into UEFI shell
5) fs0:
6) sas2flash.efi -o -f 2118it.bin
7) sas2flash.efi -o -sasadd 500605b003e321f0	<different per card>
8) sas2flash.efi -listall			<to verify fw updated>
```

***BIG NOTE***
Above is my own crib notes, but there's lots more info out there like this URL and others. This was for my Supermicro X9SCM-F mobo that had UEFI for instance...yours maybe be slightly different process and might not need to deal with UEFI.

http://brycv.com/blog/2012/flashing-it-firmware-to-lsi-sas9211-8i/


*2) Dload and use LSI's FreeBSD v15 driver.*

This one is easy. 


```
1) Dload the 'Free_BSD_Driver_P15' and copy the mpslsi.ko to /boot/kernel/.
2) Add mpslsi_load=â€YESâ€ to /boot/loader.conf
3) Reboot and run sysctl -a | grep mps ... if you see something like the below - SUCCESS - both card + system using the v15 drivers.

dev.mpslsi.1.firmware_version: 15.00.00.00
dev.mpslsi.1.driver_version: 15.00.00.00
```

Hope this helps.

My system has been running good since doing this outside of the occasional hardware gremlin...


----------



## Migelo (Jan 18, 2013)

Thanks! I'm going to try and do it now.

One thing though, after this the card will be in IT-mode, does that mean that I'll be able to switch from mps and mpi diver?


----------



## thethirdnut (Jan 18, 2013)

Do you mean the kernel's default 'mps' driver and LSI's 'mpslsi' driver?

If so, yes...just don't add anything to /boot/loader.conf and you'll be using the stock driver.

If there is an actual 'mpi' driver - I don't know what that is...once the card's are flashed only way to get them back to other state is to re-flash them.


----------



## Migelo (Jan 18, 2013)

I meant mfi driver people were mentioning before.


----------



## thethirdnut (Jan 18, 2013)

Haven't used or heard of it before - sorry.


----------



## Migelo (Jan 18, 2013)

No problem, you helped a lot, those instructions are really worth a million, saved me quite some reading.

mfi(4)


----------



## thethirdnut (Jan 18, 2013)

No problem - good luck.

Keep in mind like I said that quite a few people have had issues, etc getting the right combo of tools to do the flashing. This worked for me so hopefully helps you as well.


----------



## Migelo (Jan 18, 2013)

Did you use the 

```
megarec
```
command there because you have a UEFI?

I've tried on two PCs with the bootable DOS usb and I still can't get the sas2flsh utility to recognise my card....


----------



## Migelo (Jan 18, 2013)

Turns out those megarec commands are crucial xD

The card work flawlessly now under Windows 7 now I just need to put those drivers to /boot/kernel and I'm (hopefully) done.

And if someone is looking where to get the megarec file, check out this blog post, under "How to cross flash".


----------



## thethirdnut (Jan 18, 2013)

Good to hear you got it working.


----------



## m6tt (Jan 19, 2013)

The built-in drivers are fine, don't use LSI's drivers for this card. If you have any problems flashing the card, try a different machine (or barring that an older/newer bios) it's quite picky. My desktop worked until a recent BIOS update, I had to use some old machine to flash it this time.


----------



## Migelo (Jan 19, 2013)

It's doing now, great performance, so I'm not going to change anything.


----------



## Boeri (Mar 18, 2013)

I'm also using the IBM M1015 card (flashed in IT mode) with the latest mpslsi driver and firmware from LSI on FreeBSD 9.1. 


```
dev.mpslsi.0.%desc: LSI SAS2008
dev.mpslsi.0.%driver: mpslsi
dev.mpslsi.0.%location: slot=0 function=0
dev.mpslsi.0.%pnpinfo: vendor=0x1000 device=0x0072 subvendor=0x1000 subdevice=0x3020 class=0x010700
dev.mpslsi.0.%parent: pci1
dev.mpslsi.0.debug_level: 4
dev.mpslsi.0.disable_msix: 0
dev.mpslsi.0.disable_msi: 0
[B]dev.mpslsi.0.firmware_version: 15.00.00.00
dev.mpslsi.0.driver_version: 15.00.00.00[/B]
dev.mpslsi.0.io_cmds_active: 10
dev.mpslsi.0.io_cmds_highwater: 291
dev.mpslsi.0.chain_free: 2048
dev.mpslsi.0.chain_free_lowwater: 2014
dev.mpslsi.0.max_chains: 2048
dev.mpslsi.0.chain_alloc_fail: 0
```

One of the drives in my striped mirror zpool configuration is bad and this freezes the whole zfs pool. 


```
mpslsi0: mpssas_scsiio_timeout checking sc 0xffffff8000862000 cm 0xffffff80008c1f48
mpslsi0: mpssas_alloc_tm freezing simq
mpslsi0: timedout cm 0xffffff80008c1f48 allocated tm 0xffffff8000879908
mpslsi0: mpssas_scsiio_timeout checking sc 0xffffff8000862000 cm 0xffffff800089d708
mpslsi0: queued timedout cm 0xffffff800089d708 for processing by tm 0xffffff8000879908
mpslsi0: mpssas_scsiio_timeout checking sc 0xffffff8000862000 cm 0xffffff80008a0670
mpslsi0: mpssas_free_tm releasing simq
mpslsi0: mpssas_alloc_tm freezing simq
mpslsi0: timedout cm 0xffffff80008a0670 allocated tm 0xffffff8000879a50
mpslsi0: mpssas_free_tm releasing simq
```


```
048 scsi 0 state c xfer(noperiph:mpslsi0:0:3:0): SMID 15 abort TaskMID 89 status 0x0 code 0x0 count 1
(da3:mpslsi0:0:3:0): WRITE(10). CDB: 2a 0 2 d4 40 80 0 1 0 0
(da3:mpslsi0:0:3:0): CAM status: Command timeout
(da3:mpslsi0:0:3:0): Retrying command
(da3:mpslsi0:0:3:0): WRITE(10). CDB: 2a 0 2 d4 41 80 0 1 0 0 length 131072 SMID 325 command timeout cm 0xffffff800088f068 ccb 0xfffffe0008f74000
(da3:mpslsi0:0:3:0): WRITE(10). CDB: 2a 0 2 d4 41 80 0 1 0 0 length 131072 SMID 325 completed timedout cm 0xffffff800088f068 ccb 0xfffffe0008f74000 during recovery ioc 8048 scsi 0 state c xfe(noperiph:mpslsi0:0:3:0): SMID 16 abort TaskMID 325 status 0x0 code 0x0 count 1
(da3:mpslsi0:0:3:0): WRITE(10). CDB: 2a 0 2 d4 41 80 0 1 0 0
(da3:mpslsi0:0:3:0): CAM status: Command timeout
(da3:mpslsi0:0:3:0): Retrying command
```

I don't want that my zfs "freezes" when one of the disks is behaving bad in a striped mirror configuration. I hoped that the zfs code and mpslsi driver would drop the disk, but this seems not to be the case. 

This was the status of the pool, a few seconds before the "crash"/stuck state of the lsi driver.


```
root@freebsd-san:/root # zpool status
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 0h10m with 0 errors on Mon Mar 18 09:51:52 2013
config:

        NAME             STATE     READ WRITE CKSUM
        tank             ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            label/disk1  ONLINE       0     0     0
            label/disk2  ONLINE       0     0     3
          mirror-1       ONLINE       0     0     0
            label/disk3  ONLINE       0     0     0
            label/disk4  ONLINE       0     0     0
        cache
          label/disk5    ONLINE       0     0     0

errors: No known data errors
```


```
root@freebsd-san:/root # camcontrol stop da3
Error received from stop unit command
```

Any zfs command seems to be stuck, in a blocked state, probably due to the "crashed" lsi driver. 

Any ideas? I can try with the "normal" mps driver, but this is based on phase 14 if *I*'m correct. My firmware is phase 15, I hope this is not an issue...



*edit:*
After a long time, the zfs command succeeded. 




```
(da3:mpslsi0:0:3:0): WRITE(10). CDB: 2a 0 2 d4 4f c5 0 1 0 0
(da3:mpslsi0:0:3:0): CAM status: Command timeout
(da3:mpslsi0:0:3:0): Error 5, Periph was invalidated
(da3:mpslsi0:0:3:0): oustanding 0
(da3:mpslsi0:0:3:0): removing device entry
```


```
root@freebsd-san:/root # zpool status
  pool: tank
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub in progress since Mon Mar 18 10:52:43 2013
        2.23G scanned out of 35.3G at 18.3M/s, 0h30m to go
        0 repaired, 6.33% done
config:

        NAME                      STATE     READ WRITE CKSUM
        tank                      DEGRADED     0     0     0
          mirror-0                DEGRADED     0     0     0
            label/disk1           ONLINE       0     0     0
            11585224050511345959  REMOVED      0     0     0  was /dev/label/disk2
          mirror-1                ONLINE       0     0     0
            label/disk3           ONLINE       0     0     0
            label/disk4           ONLINE       0     0     0
        cache
          label/disk5             ONLINE       0     0     0

errors: No known data errors
```

This took longer than 5 minutes.  60 seconds timeout (kern.cam.da.default_timeout: 60) and 4x retry. Is this possible? 
Or is the mpslsi driver handling the timeouts in a different way ?


----------

