# How do I install a commercial driver?



## mariourk (Mar 4, 2013)

I have a new server, with this card.

I downloaded the FreeBSD-driver and copied the amd64-version of mpslsi.ko, meant for FreeBSD 9.0, to /boot/kernel/ and added mpslsi_load="YES" to /boot/loader.conf. However, the FreeBSD release it's actually running on, is 9.1.

It seems to work, but I'm not at all confident. What is the proper way to do this?


----------



## Carpetsmoker (Mar 4, 2013)

There is also a mps driver available in the base. This seems to be the same driver? They have the same source files, and doing a quick diff there do seem to be differences, but it seems that the version in FreeBSD base is newer / more up to date.

In any case, source is provided in the download. So you can just compile your own version if you wish to use it.

(As a bonus hint, you can just use kldload(8) on any file, ie. `# kldload mymodule.ko`).


----------



## mariourk (Mar 4, 2013)

> So you can just compile your own version if you wish to use it.


If it were Linux, I'd know how to do that. But I'm faily new to FreeBSD and I have no idea what's the FreeBSD way to do that... :\

The reason for trying this driver is that the server sometimes freezes for several seconds. Or the the networkconnections, anyway. The system simply doesn't respond for a few seconds and that it returns to normal, like nothing happened. I cannot figure out why it does that. So I decided to give this official driver a try. Until now the problem hasn't occured yet.


----------



## Carpetsmoker (Mar 4, 2013)

Normally you can just type make, but since this particular module shadows a module with the same name in the base system it seems to be a bit more involved.

Here's what I did:


```
[/home/martin/Free_BSD_Driver_P15]# rm -r /usr/src/sys/modules/mps /usr/src/sys/dev/mps/
[/home/martin/Free_BSD_Driver_P15]# tar xf mpslsi-source-15.00.00.00.tar.gz
[/home/martin/Free_BSD_Driver_P15]# mv mpslsi-source-15.00.00.00/sys/* /usr/src/sys/
[/home/martin/Free_BSD_Driver_P15]# cd /usr/src/sys/modules/mps/        
[/usr/src/sys/modules/mps]# make
[... Make output snipped ... ]
[/usr/src/sys/modules/mps]# kldload ./mpslsi.ko
[/usr/src/sys/modules/mps]# kldstat
Id Refs Address            Size     Name
 1   24 0xffffffff80200000 1323420  kernel
 2    2 0xffffffff81612000 27f8     vboxnetflt.ko
 3    2 0xffffffff81615000 87b2     netgraph.ko
 4    2 0xffffffff8161e000 30020    vboxdrv.ko
 5    1 0xffffffff8164f000 1579     ng_ether.ko
 6    1 0xffffffff81651000 3f20     vboxnetadp.ko
 7    1 0xffffffff81655000 1f445    linux.ko
 8    1 0xffffffff81675000 64a50    radeon.ko
 9    1 0xffffffff816da000 1391f    drm.ko
10    1 0xffffffff81784000 1b8a4    mpslsi.ko
```

You probably don't *need* to remove the directories from /usr/src/, you can also edit the Makefile or somesuch. This seemed the quicker answer


----------



## mariourk (Mar 5, 2013)

Ok, I managed to compile the driver and use that mpslsi.ko. However, during boot, I see a lot of this rolling over the screen. I see it again when I check if the driver is loaded with [cmd=]sysctl -a | grep mps[/cmd]


```
(probe1020:mpslsi0:0:1021:0): CAM status: Invalid Target ID
(probe1020:mpslsi0:0:1021:0): Error 22, Unretryable error
(probe1020:mpslsi0:0:1021:0): INQUIRY. CDB: 12 0 0 0 24 0 
(probe1020:mpslsi0:0:1021:0): CAM status: Invalid Target ID
(probe1020:mpslsi0:0:1021:0): Error 22, Unretryable error
(probe1021:mpslsi0:0:1022:0): INQUIRY. CDB: 12 0 0 0 24 0 
(probe1021:mpslsi0:0:1022:0): CAM status: Invalid Target ID
(probe1021:mpslsi0:0:1022:0): Error 22, Unretryable error
(probe1021:mpslsi0:0:1022:0): INQUIRY. CDB: 12 0 0 0 24 0 
(probe1021:mpslsi0:0:1022:0): CAM status: Invalid Target ID
(probe1021:mpslsi0:0:1022:0): Error 22, Unretryable error
(probe1022:mpslsi0:0:1023:0): INQUIRY. CDB: 12 0 0 0 24 0 
(probe1022:mpslsi0:0:1023:0): CAM status: Invalid Target ID
(probe1022:mpslsi0:0:1023:0): Error 22, Unretryable error
(probe1022:mpslsi0:0:1023:0): INQUIRY. CDB: 12 0 0 0 24 0 
(probe1022:mpslsi0:0:1023:0): CAM status: Invalid Target ID
(probe1022:mpslsi0:0:1023:0): Error 22, Unretryable error
```

Right after these errors, I see this

```
ses0 at mpslsi0 bus 0 scbus0 target 20 lun 0
da0 at mpslsi0 bus 0 scbus0 target 8 lun 0
da1 at mpslsi0 bus 0 scbus0 target 9 lun 0
da2 at mpslsi0 bus 0 scbus0 target 10 lun 0
da4 at mpslsi0 bus 0 scbus0 target 12 lun 0
da5 at mpslsi0 bus 0 scbus0 target 13 lun 0
da3 at mpslsi0 bus 0 scbus0 target 11 lun 0
da7 at mpslsi0 bus 0 scbus0 target 15 lun 0
da8 at mpslsi0 bus 0 scbus0 target 16 lun 0
da9 at mpslsi0 bus 0 scbus0 target 17 lun 0
da6 at mpslsi0 bus 0 scbus0 target 14 lun 0
da10 at mpslsi0 bus 0 scbus0 target 18 lun 0
da11 at mpslsi0 bus 0 scbus0 target 19 lun 0
dev.mpslsi.0.%desc: LSI SAS2308
dev.mpslsi.0.%driver: mpslsi
dev.mpslsi.0.%location: slot=0 function=0
dev.mpslsi.0.%pnpinfo: vendor=0x1000 device=0x0087 subvendor=0x1000 subdevice=0x3030 class=0x010700
dev.mpslsi.0.%parent: pci3
dev.mpslsi.0.debug_level: 4
dev.mpslsi.0.disable_msix: 0
dev.mpslsi.0.disable_msi: 0
dev.mpslsi.0.firmware_version: 15.00.00.00
dev.mpslsi.0.driver_version: 15.00.00.00
dev.mpslsi.0.io_cmds_active: 0
dev.mpslsi.0.io_cmds_highwater: 80
dev.mpslsi.0.chain_free: 2048
dev.mpslsi.0.chain_free_lowwater: 2030
dev.mpslsi.0.max_chains: 2048
dev.mpslsi.0.chain_alloc_fail: 0
```

I have no idea what these errors mean and if they are a problem. Because, everything seems to work fine.


----------



## gkontos (Mar 5, 2013)

Same errors here with WD Reds. An upgrade to 9.1-STABLE fixed them.


----------



## mariourk (Mar 5, 2013)

This server is running 9.1-RELEASE. I don't know if that makes any difference.


----------



## fonz (Mar 5, 2013)

mariourk said:
			
		

> This server is running 9.1-RELEASE. I don't know if that makes any difference.


9-STABLE is a development branch, so it gets fixes (and sometimes new bugs ). 9.1-RELEASE hardly gets anything except important security patches. The handbook explains how to track a development branch, see for yourself whether it's worth for you to try.


----------



## mariourk (Mar 5, 2013)

In that case, I'll stick with 9.1-RELEASE. It seems like the smart choice for a production server.


----------



## gkontos (Mar 5, 2013)

mariourk said:
			
		

> In that case, I'll stick with 9.1-RELEASE. It seems like the smart choice for a production server.



The smart choice really depends on your hardware. We are sharing the same problem and I presented you with a solution that solved it.

My approach to the problem was:


Use FreeBSD 9.1 with driver 14. 
Use FreeBSD 9.1 with driver 15. (From LSI)
Use FreeBSD 9.1-STABLE

The server is a SuperMicro driving a 45 drive JBOD chassis enclosure. In all cases the system was tested with 11 drives on a RAIDz3 pool.


----------



## mariourk (Mar 5, 2013)

Actually, I tried to move to 9.1-STABLE, but that didn't work.

```
freebsd-update -r 9.1-STABLE upgrade
```
I suppose I'm doing something horribly wrong 

What is the proper way to move from 9.1-RELEASE to 9.1-STABLE?


----------



## Carpetsmoker (Mar 5, 2013)

mariourk said:
			
		

> Actually, I tried to move to 9.1-STABLE, but that didn't work.
> 
> ```
> freebsd-update -r 9.1-STABLE upgrade
> ...



-STABLE is not a `release', it's a branch in svn/CVS, from this -STABLE branch new -RELEASE branches are created, and from these -RELEASE branches distribution sets, install CDs, etc. are created.

With time on the x-axis:

```
------------ CURRENT/HEAD -------->
   \
    \-> 9-STABLE  -------------------------->
              \                    \
               \-> 9.0-RELEASE      \-> 9.1-RELEASE
```

freebsd-update only works with -RELEASE, so you'll need to install from source, this is not very hard and is described in the handbook (in particular, section 25.6 and 25.7).

Once you have the source, /usr/src/Makefile also describers the basic commands which may be useful as a quick terse reference:

```
# For individuals wanting to upgrade their sources (even if only a
# delta of a few days):
#
#  1.  `cd /usr/src'       (or to the directory containing your source tree).
#  2.  `make buildworld'
#  3.  `make buildkernel KERNCONF=YOUR_KERNEL_HERE'     (default is GENERIC).
#  4.  `make installkernel KERNCONF=YOUR_KERNEL_HERE'   (default is GENERIC).
#       [steps 3. & 4. can be combined by using the "kernel" target]
#  5.  `reboot'        (in single user mode: boot -s from the loader prompt).
#  6.  `mergemaster -p'
#  7.  `make installworld'
#  8.  `make delete-old'
#  9.  `mergemaster'    (you may wish to use -i, along with -U or -F).
# 10.  `reboot'
# 11.  `make delete-old-libs' (in case no 3rd party program uses them anymore)
```


----------



## mariourk (Mar 5, 2013)

As someone who is new to FreeBSD, I think I'll stick with REALEASE for a while. Especially for the production-servers... :\

Despite having these errors, the system itself shows no problems and runs fine.

Could someone explain what these errors exactly mean? And how serious they are?


----------



## Carpetsmoker (Mar 5, 2013)

> Could someone explain what these errors exactly mean? And how serious they are?





> (probe1020:mpslsi0:0:1021:0): INQUIRY. CDB: 12 0 0 0 24 0
> (probe1020:mpslsi0:0:1021:0): CAM status: Invalid Target ID
> (probe1020:mpslsi0:0:1021:0): Error 22, Unretryable error



The SCSI INQUIRY command gets basic info from the device, in this case the command fails because the target ID 12 is considered invalid for whatever reason.

What this *exactly* means or why this is caused is difficult to say. It *may* be a symptom of a more serious problem, or it may not be.
If you want a definitive answer, then a deeper investigation into the driver is required. This is something that's outside the scope of these forums. There are a number of FreeBSD mailing lists such as *freebsd-scsi@* where people with more specific knowledge of this subject hang out.


----------



## mariourk (Mar 5, 2013)

> in this case the command fails because the target ID 12 is considered invalid for whatever reason.


This server has 12 SAS disks. da0 to da11. I suppose that's the reason because ID 12 is invalid? And that it's not a big deal?


----------



## Carpetsmoker (Mar 5, 2013)

mariourk said:
			
		

> This server has 12 SAS disks. da0 to da11. I suppose that's the reason because ID 12 is invalid? And that it's not a big deal?



This *may* be the case, or it *may* be something else (such as trying to get the status of an existing disk). You can use *camcontrol devlist* to find out about your disks and which LUN they have.


----------



## mariourk (Mar 5, 2013)

That gives me this

```
<SEAGATE ST33000650SS 0004>        at scbus0 target 8 lun 0 (da0,pass0)
<SEAGATE ST33000650SS 0004>        at scbus0 target 9 lun 0 (da1,pass1)
<SEAGATE ST3300657SS 0008>         at scbus0 target 10 lun 0 (da2,pass2)
<SEAGATE ST3300657SS 0008>         at scbus0 target 11 lun 0 (da3,pass3)
<SEAGATE ST33000650SS 0004>        at scbus0 target 12 lun 0 (da4,pass4)
<SEAGATE ST33000650SS 0004>        at scbus0 target 13 lun 0 (da5,pass5)
<SEAGATE ST33000650SS 0004>        at scbus0 target 14 lun 0 (da6,pass6)
<SEAGATE ST33000650SS 0004>        at scbus0 target 15 lun 0 (da7,pass7)
<SEAGATE ST33000650SS 0004>        at scbus0 target 16 lun 0 (da8,pass8)
<SEAGATE ST33000650SS 0004>        at scbus0 target 17 lun 0 (da9,pass9)
<SEAGATE ST33000650SS 0004>        at scbus0 target 18 lun 0 (da10,pass10)
<SEAGATE ST33000650SS 0004>        at scbus0 target 19 lun 0 (da11,pass11)
< 80H10323501A0 0703>              at scbus0 target 20 lun 0 (ses0,pass12)
```
All the disks are present. I have no idea what ses0 is.


----------



## gkontos (Mar 5, 2013)

@mariourk,

Please post or send me the full dmesg of the server. I strongly believe that we are facing the same bug here.

If you want to try 9.1-STABLE the procedure is pretty easy.  

@Carpetsmoker,

The issue has been raised in the mailing list(s).


----------



## mariourk (Mar 6, 2013)

This is the output of [cmd=]dmesg[/cmd]

```
[... lots of probe messages snipped, they are all the same ...]

(probe1021:mpslsi0:0:1022:0): INQUIRY. CDB: 12 0 0 0 24 0
(probe1021:mpslsi0:0:1022:0): CAM status: Invalid Target ID
(probe1021:mpslsi0:0:1022:0): Error 22, Unretryable error
(probe1021:mpslsi0:0:1022:0): INQUIRY. CDB: 12 0 0 0 24 0
(probe1021:mpslsi0:0:1022:0): CAM status: Invalid Target ID
(probe1021:mpslsi0:0:1022:0): Error 22, Unretryable error
(probe1022:mpslsi0:0:1023:0): INQUIRY. CDB: 12 0 0 0 24 0
(probe1022:mpslsi0:0:1023:0): CAM status: Invalid Target ID
(probe1022:mpslsi0:0:1023:0): Error 22, Unretryable error
(probe1022:mpslsi0:0:1023:0): INQUIRY. CDB: 12 0 0 0 24 0
(probe1022:mpslsi0:0:1023:0): CAM status: Invalid Target ID
(probe1022:mpslsi0:0:1023:0): Error 22, Unretryable error
ses0 at mpslsi0 bus 0 scbus0 target 20 lun 0
ses0: < 80H10323501A0 0703> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device
ums0: <ATEN ATEN  CS-175854, class 0/0, rev 1.10/1.00, addr 3> on usbus1
da0 at mpslsi0 bus 0 scbus0 target 8 lun 0
da0: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da1 at mpslsi0 bus 0 scbus0 target 9 lun 0
da1: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da2 at mpslsi0 bus 0 scbus0 target 10 lun 0
da2: <SEAGATE ST3300657SS 0008> Fixed Direct Access SCSI-5 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da4 at mpslsi0 bus 0 scbus0 target 12 lun 0
da4: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da5 at mpslsi0 bus 0 scbus0 target 13 lun 0
da5: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da3 at mpslsi0 bus 0 scbus0 target 11 lun 0
da3: <SEAGATE ST3300657SS 0008> Fixed Direct Access SCSI-5 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
da7 at mpslsi0 bus 0 scbus0 target 15 lun 0
da7: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da8 at mpslsi0 bus 0 scbus0 target 16 lun 0
da8: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da9 at mpslsi0 bus 0 scbus0 target 17 lun 0
da9: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da6 at mpslsi0 bus 0 scbus0 target 14 lun 0
da6: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
ums0: 5 buttons and [XYZ] coordinates ID=0
SMP: AP CPU #7 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
da10 at mpslsi0 bus 0 scbus0 target 18 lun 0
da10: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da11 at mpslsi0 bus 0 scbus0 target 19 lun 0
da11: <SEAGATE ST33000650SS 0004> Fixed Direct Access SCSI-6 device
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
Timecounter "TSC-low" frequency 8593937 Hz quality 1000
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
Root mount waiting for: usbus0
uhub4: 5 ports with 5 removable, self powered
ugen0.4: <American Megatrends Inc.> at usbus0
ukbd1: <Keyboard Interface> on usbus0
kbd2 at ukbd1
ums1: <Mouse Interface> on usbus0
ums1: 3 buttons and [Z] coordinates ID=0
Trying to mount root from ufs:/dev/mirror/gm0a [rw]...
ZFS filesystem version 5
ZFS storage pool version 28
em0: link state changed to UP
lagg0: link state changed to UP
em1: link state changed to UP
em3: link state changed to UP
em2: link state changed to UP
```


----------



## AndyUKG (Mar 6, 2013)

mariourk said:
			
		

> I have no idea what ses0 is.



It's the SAS Expander: http://en.wikipedia.org/wiki/Serial_attached_SCSI#SAS_expanders


----------



## boris_net (Jun 4, 2013)

gkontos said:
			
		

> My approach to the problem was:
> 
> 
> Use FreeBSD 9.1 with driver 14.
> ...



Just curious, since I am hitting a very similar scenario. Did you use the mps driver from 9.1 STABLE? Or did you compile LSI mpslsi drivers using 9.1 STABLE sources?

Thanks


----------



## mariourk (Jun 4, 2013)

I'm using the mps driver from 9.1 STABLE. That works fine.


----------



## gkontos (Jun 4, 2013)

boris_net said:
			
		

> Just curious, since I am hitting a very similar scenario. Did you use the mps driver from 9.1 STABLE? Or did you compile LSI mpslsi drivers using 9.1 STABLE sources?
> 
> Thanks



I tried both and I kept getting some weird errors with the LSI driver so I decided to use the driver form 9.1-STABLE.

As a side note, the machine is running 9.1-STABLE which was compiled around three months ago. There have been a lot of enhancements and bug fixes since then but we are currently unable to find a suitable window for another upgrade.


----------



## mariourk (Jun 4, 2013)

gkontos said:
			
		

> I tried both and I kept getting some weird errors with the LSI driver so I decided to use the driver form 9.1-STABLE.
> 
> As a side note, the machine is running 9.1-STABLE which was compiled around 3 moths ago. There have been a lot of enhancements and bug fixes since then but we are currently unable to find a suitable window for another upgrade.



The same applies to me. Let me know how it turns out if you do upgrade. I will do the same, if I happen to do the upgrade before you


----------



## boris_net (Jun 7, 2013)

Hi *a*ll,

What kind of version of drivers do you get from 9.1-STABLE? I am still on 14.00.00.01-fbsd.


```
uname -a
FreeBSD houdini 9.1-STABLE FreeBSD 9.1-STABLE #0 r251482: Fri Jun  7 10:21:16 EDT 2013     root@houdini:/usr/obj/usr/src/sys/NOYO  amd64
```


```
sysctl -a  | grep mps | grep version
dev.mps.0.firmware_version: 15.00.00.00
dev.mps.0.driver_version: 14.00.00.01-fbsd
dev.mps.1.firmware_version: 15.00.00.00
dev.mps.1.driver_version: 14.00.00.01-fbsd
dev.mps.2.firmware_version: 15.00.00.00
dev.mps.2.driver_version: 14.00.00.01-fbsd
```

I have also seen a 16.00.00.00 for FreeBSD on the LSI website. Has anybody tested it?

Boris


----------



## ethoms (Feb 26, 2015)

I know this is an old thread. But I'm having a similar problem after upgrding to FreeBSD 10.1. My drives become 'Removed' and my pool nearly got messed up.

I was running the mpslsi from LSI website on FreeBSD 8.3 for 2 years without any problems. NO I'm using the mps in the 10.1-RELEASE base.

After hunting around, it seems that whilst the firmware can be newer than the driver, the other way around causes major problems. Particularly with SATA drives. So basically, we need to update the firmware before updating any drivers, and it must be same versions or higher than the driver.

I will wait for my pool to finished resilvering, then flash the firmware with version 20, and use the mpslsi version 20 driver from their website. Despite that driver beeing build for FreeBSD 10.0. I'll take the risk it works on 10.1, since the ABI compatability _should_ be OK. If I have any further issues, I'll probably build it from source (again from LSI website).

My super stable 8.3 experience was with the mpslsi. I can't remember, but I think I had issue with the base mps on 8.3 as well. Or perhaps there was no mps in base at at that time, can't remember.

Oh LSI, thanks for these shananigans! I had done a perfect upgrade / migration and then I had shutdown servies for half a business day. Not to mention having a severe panic attack whilst watching my data pool break down severely.


----------



## ethoms (Feb 27, 2015)

> I will wait for my pool to finished resilvering, then flash the firmware with version 20, and use the mpslsi version 20 driver from their website. Despite that driver being build for FreeBSD 10.0. I'll take the risk it works on 10.1, since the ABI compatability _should_ be OK. If I have any further issues, I'll probably build it from source (again from LSI website).



Well I wrong about that. The latest mpslsi driver from LSI website causes the system to reboot instantly. Even when compiled from source. I guess they haven't updated it for FreeBSD 10.1 yet.

I updated my firmware to version 20.0.2.0 and the mps in 10.1-RELEASE base is based on LSI's 19.0 driver version.I still get the SCSI / CAM errors. Although not yet had a drive dropped out.

It appears to be  probleM with SATA drives and SAS expanders (JBOD), with regards to timeouts. Perhaps it's also related to my WD black's firmware.

At the moment I'm pretty screwed, the data pool is in production AND it could bail out any minute. I am too tired to continue, after 2 days with no sleep. It's crazy! Both my hba card (LSI SAS 9200-8e) and my JBOB array are from LSI. The later being rebranded by Supermicro. Both products are supposed to support SATA, and yet it clearly got MAJOR stability issues. It plain doesn't work. I was told LSI was good for FreeBSD.


----------



## kpa (Feb 27, 2015)

ethoms said:


> Well I wrong about that. The latest mpslsi driver from LSI website causes the system to reboot instantly. Even when compiled from source. I guess they haven't updated it for FreeBSD 10.1 yet.



The problem is that unlike the stable API/ABI for userland software, FreeBSD does not have a stable KPI/KBI (kernel programming/binary interface). This means the hardware vendor who wants to provide FreeBSD drivers for their products has to update their drivers for every single minor release of FreeBSD. Commercial OSes such as Windows do have stable KPI/KBI and that's why you can expect that a driver for let's says Windows 7 will work on Windows 7 and even on Windows 8 even after hundreds of updates to the OS.


----------



## ethoms (Feb 27, 2015)

Thanks kpa, that's interesting. It amazes me how well Microsoft have done at keeping Windows API/ABI compatibility for so long. On the other hand, I will never willingly use it again, for anything. Unices all the way, despite any shortfalls.

I managed to resolve my HBA controller issues. It was the firmware version after all.

Thanks to *Borja Marcos *email (https://lists.freebsd.org/pipermail/freebsd-scsi/2014-October/006506.html) in some FreeBSD mailing list, it was indicated that downgrading to version P19 solves the CAM / SCSI errors and timeouts.

So to recap, my problems started by upgrading to 10.1-RELEASE, using the base mps driver (based on the LSI P19 driver). At that time I was on firmware version P16. Then I upgraded to firmware version P20 and still had the same problems. Only after downgrading to P19 did my problems go away instantly. It's also worth noting that the following settings in /boot/loader.conf help with SATA drives in SAS arrays:

```
# Change I/O queue settings to play nice with SATA NCQ and
# other storage controller features.
vfs.zfs.vdev.min_pending="1"
vfs.zfs.vdev.max_pending="1"
```
Also not that when flashing the LSI firmware, do a `sasflsh -o -e 6` to erase the existing firmware and NVDATA. This means using the DOS flash util, not the FreeBSD one. I recommend using a FreeDOS USB image, found here:  http://chtaube.eu/computers/freedos/bootable-usb/
After using dd to load the image on your USB memstick, copy the sasflsh.exe file and the firmware ROM file to the USB drive. Boot into it and perform the erase, then the flash. Also note that many modern Intel chipsets will give an error when trying to use sasflsh. I had to move the HBA controller card to another machine to perform the erase/flash.

I'm not 100% convinced the controller is working perfectly now, but it's made a huge difference at least. Time will tell. I'll try to report back after a few weeks of use.


----------



## ethoms (Mar 5, 2015)

To follow my issue, which is still ongoing, you may refer to another thread: https://forums.freebsd.org/threads/what-happened-to-vfs-zfs-vdev-max_pending.50591/.

In summary, currently it looks like my issue may be due to mixing SAS and SATA drives in the same enclosure. Somehow the above settings (workaround) in /boot/loader.conf made me get away with it on FreeBSD 8.3.

However, I'm unsure if a similar trick, by turning sata ncq queueing down to 1 via `camcontrol tags da? -N 1` is also helping solve the issue.


----------

