# Storage management Software for a LSI 1064e



## minimike (Nov 16, 2010)

Hi

For the LSI 1064E and LSI 1068 SAS Controllers, is there any Software for FreeBSD to monitor and configure the Storage available? LSI offers the LSI MegaRaid Storage Manager only for Windows and Linux.

best regards
Darko


----------



## Terry_Kennedy (Nov 17, 2010)

minimike said:
			
		

> For the LSI 1064E and LSI 1068 SAS Controllers, is there any Software for FreeBSD to monitor and configure the Storage available? LSI offers the LSI MegaRaid Storage Manager only for Windows and Linux.



Look at /usr/sbin/mptutil. It doesn't have all of the functionality of the Windows version. In particular, it doesn't decode controller events, so `# mptutil show events` gives you a hex dump instead of useful messages.


```
(0:83) new-gate:/tmp# mptutil show adapter
mpt0 Adapter:
       Board Name: SAS6IR
   Board Assembly: 
        Chip Name: C1068E
    Chip Revision: UNUSED
      RAID Levels: RAID0, RAID1, RAID1E
    RAID0 Stripes: 64K
   RAID1E Stripes: 64K
 RAID0 Drives/Vol: 2-10
 RAID1 Drives/Vol: 2
RAID1E Drives/Vol: 3-10
(0:84) new-gate:/tmp# mptutil show volumes
mpt0 Volumes:
  Id     Size    Level   Stripe  State  Write-Cache  Name
     0 (  279G) RAID-1          OPTIMAL   Enabled   
(0:85) new-gate:/tmp# mptutil show drives
mpt0 Physical Drives:
   0 (  279G) ONLINE <WDC WD3000HLFS-0 4V02> SATA bus 0 id 1
   1 (  279G) ONLINE <WDC WD3000HLFS-0 4V02> SATA bus 0 id 9
```


----------



## minimike (Nov 23, 2010)

Thank you very much for your answer. On FreeBSD mptutil is smaller and not so heavy like LSI MSM on Linux. It doesn't need Java and it's simply lightwight.
Is there something for FreeBSD to use it with Nagios or anything else for network based monitoring?

best regards
Darko


----------



## danstoner (Dec 6, 2010)

No experience personally, but I found a similar post and there is apparently another utility named "lsiutil" from the LSI web site that might be an option for you.

http://lists.us.dell.com/pipermail/linux-poweredge/2009-March/038764.html

Any of these things that can be scripted ought to be usable by nagios in some way.


----------



## Terry_Kennedy (Dec 7, 2010)

danstoner said:
			
		

> No experience personally, but I found a similar post and there is apparently another utility named "lsiutil" from the LSI web site that might be an option for you.
> 
> http://lists.us.dell.com/pipermail/linux-poweredge/2009-March/038764.html



There's no native FreeBSD version in the lsiutil archive, and the Linux version wants some special devices not provided by FreeBSD's Linux emulation:


```
(0:200) new-gate:~terry# ./lsiutil

LSI Logic MPT Configuration Utility, Version 1.60, July 11, 2008
sh: /sbin/modprobe: No such file or directory
/bin/mknod: `/dev/mptctl': No such file or directory
Couldn't open /dev/mptctl or /dev/mpt2ctl!

0 MPT Ports found
```
There's a source tarball in there as well, but it doesn't compile out-of-the-box on FreeBSD and making it work may or may not be possible, depending on whether it needs control interfaces not exposed by the FreeBSD mpt driver.

The mpt-status package linked from your URL also relies on the Linux mptctl interface.


----------



## bsdnet (Jan 15, 2015)

Any folks got luck on this? on 10.1-RELEASE, `mptutil` just segmentation fault.

```
# mptutil show drives
Segmentation fault (core dumped)
# mptutil show config
Segmentation fault (core dumped)
# uname -a
FreeBSD StorOS 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014  root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```


----------



## Terry_Kennedy (Jan 15, 2015)

bsdnet said:


> Any folks got luck on this? on 10.1-RELEASE, `mptutil` just segmentation fault.
> 
> ```
> # mptutil show drives
> ...


I have an mpt(4) adapter on FreeBSD 10.1 amd64 (r276774 from 10-STABLE) and while I don't have any drives connected to it, mptutil(8) doesn't segfault:

```
(0:17) test2:/sysprog/terry# mptutil show adapter
mpt0 Adapter:
  Board Name: 0
  Board Assembly: 0
  Chip Name: C1030
  Chip Revision: 0
  RAID Levels: none
(0:18) test2:/sysprog/terry# mptutil show config
mpt0 Configuration: 0 volumes, 0 drives
(0:19) test2:/sysprog/terry# mptutil show drives
mpt0 Physical Drives:
```


----------



## bsdnet (Jan 15, 2015)

Thanks Terry, 
I am wondering what's going wrong here. Here is the output of  `pciconf -lv` and `dmesg`

```
mpt0@pci0:4:0:0:   class=0x010000 card=0x31401000 chip=0x00581000 rev=0x08 hdr=0x00
  vendor  = 'LSI Logic / Symbios Logic'
  device  = 'SAS1068E PCI-Express Fusion-MPT SAS'
  class  = mass storage
  subclass  = SCSI

 # dmesg | grep  -i MPT
mpt0: <LSILogic SAS/SATA Adapter> port 0xe000-0xe0ff mem 0xfba10000-0xfba13fff,0xfba00000-0xfba0ffff irq 27 at device 0.0 on pci4
mpt0: MPI Version=1.5.19.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 0 Active Volumes (2 Max)
mpt0: 0 Hidden Drive Members (14 Max)
da0 at mpt0 bus 0 scbus0 target 40 lun 0
da1 at mpt0 bus 0 scbus0 target 41 lun 0
da2 at mpt0 bus 0 scbus0 target 42 lun 0
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da3 at mpt0 bus 0 scbus0 target 43 lun 0
da6 at mpt0 bus 0 scbus0 target 46 lun 0
da7 at mpt0 bus 0 scbus0 target 47 lun 0
da4 at mpt0 bus 0 scbus0 target 44 lun 0
da5 at mpt0 bus 0 scbus0 target 45 lun 0
pid 914 (mptutil), uid 0: exited on signal 11 (core dumped)
pid 915 (mptutil), uid 0: exited on signal 11 (core dumped)
```

Any clue? Maybe I need download the source to debug this.


----------



## Oko (Jan 15, 2015)

minimike said:


> Hi
> 
> For the LSI 1064E and LSI 1068 SAS Controllers, is there any Software for FreeBSD to monitor and configure the Storage available? LSI offers the LSI MegaRaid Storage Manager only for Windows and Linux.
> 
> ...



I am running mptutil on 

10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014

without any troubles. 

Is this the kind a hardware you are talking about

http://hwraid.le-vert.net/wiki/LSIFusionMPT

That is crappy hardware. For high end RAID controllers you do have  sysutils/megarc. For crappy once there is  sysutils/sas2ircu 

The be frank with you. If I had that crap I would see if you could switch into JBOD mode and use individual disks with ZFS. One thing to watch is definitely the quality of the rest of the hardware and amount of RAM. Please see my infamous thread ZFS vs HAMMER.

Which file system are you going to use if you are going to go with HW Raid? Are you going to use UFS (FreeBSD has nothing else). I am using UFS on that server which have fully functional mptutil due to the fact that hardware is not good enough for ZFS and I don't use Linux for data storage. You might be better of using free clone of Red Hat for which you have proprietary monitoring tool and XFS.

Another option is using HAMMER of DragonFly with HW Raid but than again you are running into the problem of monitoring HW Raid controller.


----------



## bsdnet (Jan 15, 2015)

Oko said:


> Is this the kind a hardware you are talking about
> 
> http://hwraid.le-vert.net/wiki/LSIFusionMPT
> 
> That is crappy hardware. For high end RAID controllers you do have  sysutils/megarc. For crappy once there is  sysutils/sas2ircu



Yes, that is the hardware I have. Whether it is crappy or not, I still need to live with it.  
Thanks, will try your suggestion out. 



Oko said:


> The be frank with you. If I had that crap I would see if you could switch into JBOD mode and use individual disks with ZFS. One thing to watch is definitely the quality of the rest of the hardware and amount of RAM. Please see my infamous thread ZFS vs HAMMER.
> 
> Which file system are you going to use if you are going to go with HW Raid? Are you going to use UFS (FreeBSD has nothing else). I am using UFS on that server which have fully functional mptutil due to the fact that hardware is not good enough for ZFS and I don't use Linux for data storage. You might be better of using free clone of Red Hat for which you have proprietary monitoring tool and XFS.
> 
> Another option is using HAMMER of DragonFly with HW Raid but than again you are running into the problem of monitoring HW Raid controller.



My card was already in JBOD mode. For filesystem,  haven't decided yet. ZFS maybe cost a lot of RAM, but it has been proved in enterprise. Another filesystem? Unless I created my startup, and write it from scratch. Or else, does not want to test another one. I am very conservative in technology consuming.
See how many filesystems Linux claims it supports? and how many are really used in enterprise?  My daily job is still using ext2.

Another BSD flavor? We have too many! If one can be the best, that is enough.


----------



## Terry_Kennedy (Jan 15, 2015)

bsdnet said:


> Any clue? Maybe I need download the source to debug this.


Does it segfault if you invoke it with no arguments? How about as `mptutil version`? If those segfault, I'd suspect something is wrong with either the libraries on your system or with the prebuilt mptutil(8) binary.

In any case, a `# ktrace mptutil; kdump` would be informative - it will let us know how far it gets before it segfaults.


----------



## Terry_Kennedy (Jan 15, 2015)

Oko said:


> Is this the kind a hardware you are talking about
> 
> http://hwraid.le-vert.net/wiki/LSIFusionMPT
> 
> That is crappy hardware.


The mpt(4) driver encompasses a very large range of controller models. In my case, it is a PCI Express SCSI (traditional Ultra320 parallel SCSI, not SAS) host adapter without RAID features.

I've been very happy with all relatively recent (last decade or so) LSI controllers. Most of the ones I've used have been Dell-branded and in Dell servers. In some cases I'm using controller-based RAID (normally RAID 5 w/ hot spare, sometimes simple mirroring) and in other cases I'm exporting the drives as individual volumes for ZFS usage.

I'm not going to pick on other brands here, but you could do far worse than LSI products.


----------



## bsdnet (Jan 15, 2015)

Hi Terry,

`mptutil version` and `mptutil show adapter` looks good.
I use the GDB to get the call trace and can't get it -- because :

```
ccb->ccb_h.status =1;
CAM_STATUS_MASK = 0x3F
CAM_REQ_CMP = 0x01
```
Some compiler problem? 
	
	



```
cam_freeccb(ccb);
```
is not supposed to be called.


```
(gdb) b
Breakpoint 1 at 0x40253a: file mpt_cam.c, line 292.
(gdb) bt
#0  0x0000000800f0c24c in sbrk () from /lib/libc.so.7
#1  0x0000000800f0c7af in sbrk () from /lib/libc.so.7
#2  0x0000000800f17df5 in free () from /lib/libc.so.7
#3  0x000000000040253a in mpt_fetch_disks (fd=<value optimized out>,
  ndisks=0x7fffffffe9bc, disksp=0x7fffffffe9c0) at mpt_cam.c:292
#4  0x0000000000405ce9 in show_drives (ac=<value optimized out>,
  av=<value optimized out>) at mpt_show.c:500
#5  0x000000000040192c in main (ac=2, av=0x7fffffffea98) at mptutil.c:117
(gdb) frame 3
#3  0x000000000040253a in mpt_fetch_disks (fd=<value optimized out>,
  ndisks=0x7fffffffe9bc, disksp=0x7fffffffe9c0) at mpt_cam.c:292
292       cam_freeccb(ccb);
(gdb) l
287       cam_freeccb(ccb);
288       return (error);
289     }
290
291     if ((ccb->ccb_h.status & CAM_STATUS_MASK) != CAM_REQ_CMP) {
292       cam_freeccb(ccb);
293       return (EIO);
294     }
295     cam_freeccb(ccb);
296
(gdb) p ccb->ccb_h.status
$3 = 1
```

Not sure how to file a bug yet. The output of `kdump`you asked is attached.


----------



## Terry_Kennedy (Jan 15, 2015)

bsdnet said:


> Some compiler problem?


Something odd is going on here. mptutil is stripped by default, so you shouldn't have the symbol table. Can you tell me some more about the system you're running on? Your earlier post indicated you were running a release kernel. Was this installed from a FreeBSD distribution or a 3rd-party build? Did you compile anything yourself? If so, do you have anything at all in /etc/make.conf? Messing with compiler flags in there should really carry a "WARNING: No user serviceable parts inside" disclaimer.



> Not sure how to file a bug yet.


Go here. But I'd suggest doing a few more rounds with me first to make sure it isn't something peculiar to your install.



> The output of `kdump`you asked is attached.



This is interesting. It has successfully cycled through your first 3 dev/passN files, but segfaulted when working on /dev/pass3.

Can you reply with the output of `# camcontrol devlist -v`? Also, the full contents of /var/run/dmesg.boot might be useful. If you feel you need to censor anything in there, just xxx it out, or PM it to me if you don't want others to see it. [There normally isn't anything sensitive in there, but some people don't want to show their Ethernet addresses or any field test hardware that might be present.]


----------



## bsdnet (Jan 15, 2015)

Terry_Kennedy said:


> Something odd is going on here. mptutil is stripped by default, so you shouldn't have the symbol table. Can you tell me some more about the system you're running on? Your earlier post indicated you were running a release kernel. Was this installed from a FreeBSD distribution or a 3rd-party build? Did you compile anything yourself? If so, do you have anything at all in /etc/make.conf? Messing with compiler flags in there should really carry a "WARNING: No user serviceable parts inside" disclaimer.



My bad. Should clarify that after `mptutil` coredumped, and you showed it is working on your setup. I downloaded source code of
`mptutil` from 10.1-RELEASE, and compile it with -g.
Then I tired to run `gdb` to provide more info.

```
# uname -a
FreeBSD StorOS 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014  root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```

The output of `camcontrol devlist -v` lists all the 8 disks I plugged in.


----------



## bsdnet (Jan 17, 2015)

The output of `dmesg`and `camcontrol devlist -v` is attached.


----------



## Terry_Kennedy (Jan 17, 2015)

bsdnet said:


> The output of `dmesg`and `camcontrol devlist -v` is attached.


These are SAS drives (at least the RE4 on da3/pass3 is). I also suspect you have a SAS expander in the chassis, give the improbably high target numbers for direct attachment.

If you don't have data on your da3 drive, try removing it and seeing if the utility still segfaults (another ktrace/kdump would be useful). I'd also suggest removing all but the boot drive and seeing if you still have the problem. This assumes that the drives are in hot-swap trays - if you have to unbolt them, try my next suggestion first.

Try booting from an installation CD/DVD or memory stick and selecting the "fixit" option. This should drop you to a shell where you can see if `# mptutil` still segfaults. This assumes that it is on the installation media - I didn't check. Also, you could try older or newer FreeBSD releases to see if this is specific to 10.1 (or 10.x).


----------



## bsdnet (Jan 17, 2015)

Thanks Terry,
This is a Lenovo Server with 8 disks. Don't think it has a SAS expander.
For the root drives, I am using raiz-2 on 4 disks, which can sustain 2 drives failure.
Here is what I have tried:

One by one removed 6 hard drives, `mptutil show drives` still segfaulted
Boot from one installation memory stick with 10.1-RELEASE, switch to shell, `mptutil show drives` still segfaulted


----------



## Terry_Kennedy (Jan 17, 2015)

bsdnet said:


> Thanks Terry,
> One by one removed 6 hard drives, `mptutil show drives` still segfaulted


How about shutting down the system and removing all drives, then booting from the 10.1-RELEASE stick and see if it still faults with no drives? Then shut down and reinstall your drives. This _should_ be harmless, though I'd suggest backing up first just to be safe.



> Boot from one installation memory stick with 10.1-RELEASE, switch to shell, `mptutil show drives` still segfaulted


Ok. Why don't you try the 8.4 and current (11) images and see if both of those fault.


----------



## bsdnet (Jan 17, 2015)

Terry_Kennedy said:


> How about shutting down the system and removing all drives, then booting from the 10.1-RELEASE stick and see if it still faults with no drives? Then shut down and reinstall your drives. This _should_ be harmless, though I'd suggest backing up first just to be safe.



Without drives,  booting from USB directly,  `mptutil` does not segfault. However, when I reinstall one drives. `mptutil` will segfault.
Looks this is a bug for me, agree?


----------



## Terry_Kennedy (Jan 17, 2015)

bsdnet said:


> Without drives,  booting from USB directly,  `mptutil` does not segfault. However, when I reinstall one drives. `mptutil` will segfault.
> Looks this is a bug for me, agree?


Yes. It would be interesting to see if this behavior is the same on 8.4 and CURRENT. If it doesn't segfault on CURRENT, then there may be a fix that can be MFC'd to 10.1.


----------



## bsdnet (Jan 17, 2015)

It is a bug and should be confirmed.

```
259  if ((ccb->ccb_h.status & CAM_STATUS_MASK) != CAM_REQ_CMP) {
  260  cam_freeccb(ccb);
  261  return (EIO);
  262  }
  263  cam_freeccb(ccb);
  264
  265  /*
  266  * A last block of 2^32-1 means that the true capacity is over 2TB,
  267  * and we need to issue the long READ CAPACITY to get the real
  268  * capacity.  Otherwise, we're all set.
  269  */
  270  if (scsi_4btoul(rcap.addr) != 0xffffffff) {
  271  disk->maxlba = scsi_4btoul(rcap.addr);
  272  return (0);
  273  }
  274
  275  /* Zero the rest of the ccb. */
  276  bzero(&(&ccb->ccb_h)[1], sizeof(struct ccb_scsiio) -
  277  sizeof(struct ccb_hdr));
  278
  279  scsi_read_capacity_16(&ccb->csio, 1, NULL, MSG_SIMPLE_Q_TAG, 0, 0, 0,
  280  (uint8_t *)&rcaplong, sizeof(rcaplong), SSD_FULL_SIZE, 5000);
  281
  282  /* Disable freezing the device queue */
  283  ccb->ccb_h.flags |= CAM_DEV_QFRZDIS;
  284
  285  if (cam_send_ccb(dev, ccb) < 0) {
  286  error = errno;
  287  cam_freeccb(ccb);
  288  return (error);
  289  }
  290
  291  if ((ccb->ccb_h.status & CAM_STATUS_MASK) != CAM_REQ_CMP) {
  292  cam_freeccb(ccb);
  293  return (EIO);
  294  }
  295  cam_freeccb(ccb);
```

appears two times at line 259  and 291.
It just re-used freed pointer.  *Bug 196819* is filed to track this issue.
EDIT: I also add my fix as attachment and hope somebody can pick it up.


----------



## ralphbsz (Jan 22, 2015)

The basic issue is understood ... bug in the code.  But:



bsdnet said:


> This is a Lenovo Server with 8 disks. Don't think it has a SAS expander.


What model?  What disk configuration?  Where is the SAS HBA installed (back or front, or motherboard)?  Is there a SAS expander card installed in the front (internal) PCIe slot?

Some servers (I happen to be familiar with Lenovo ones) have SAS expanders in unexpected places.  For servers that have more than 4 disks in the front panel, there are usually SAS expanders somewhere.  I know how to locate them using Linux (lsscsi gives you a list of all of them, then you can use the /sys file system to track down the SAS connectivity, extremely tediously).  In FreeBSD, I have never tried this, and I would start with the dmesg output.

But this is just a side remark, for future clarification.


----------



## bsdnet (Jan 23, 2015)

ralphbsz said:


> The basic issue is understood ... bug in the code.  But:
> 
> 
> What model?  What disk configuration?  Where is the SAS HBA installed (back or front, or motherboard)?  Is there a SAS expander card installed in the front (internal) PCIe slot?



*ThinkServer 70B7002RUX 5U Tower Server*
This 1st gen SAS HBA is installed in PCIE slot of Motherboard. 
Two mini-SAS cable, one to upper panel; one to lower panel.
Did not see expander though initially, I thought it has one.


----------



## ralphbsz (Jan 24, 2015)

bsdnet said:


> *ThinkServer 70B7002RUX 5U Tower Server*
> This 1st gen SAS HBA is installed in PCIE slot of Motherboard.
> Two mini-SAS cable, one to upper panel; one to lower panel.
> Did not see expander though initially, I thought it has one.


Actually, that's the big tower model.  It may very well have no expander at all, because it can only handle 8 big (3.5") disks in hardware, which means the SAS HBA has enough ports to feed all disks.  I think you can buy a similar model with disk hardware for 16 small disks; that probably causes an expander to be added somewhere.  I'm not very familiar with the tower models though, only seen them from the outside, so I wouldn't know where to look for the expanders.

On the rack-mount models (x3650), the ones that can handle 16 small disks on the front panel have SAS expanders, usually mounted on a card that physically goes into the PCIe slot on the left front of the machine, and then connects to the motherboard "RAID" controller via SAS cables.  There are other options too, depending on how many SAS HBAs and how many disk holders are installed; by adding a SAS HBA in that slot and running a different cable harness, one can also feed the front-panel disks without expanders.


----------



## Terry_Kennedy (Jan 24, 2015)

ralphbsz said:


> Actually, that's the big tower model.  It may very well have no expander at all, because it can only handle 8 big (3.5") disks in hardware, which means the SAS HBA has enough ports to feed all disks.


I think there might be something funny going on, since the OP's first hard drive is at target 40. I have a SAS1068E card here with the same PCI ID and rev (chip=0x00581000 rev=0x08) as the OP, and the disk on it is target 0.

One of the options in the card's advanced BIOS menu is a display of the SAS topology. I think that's the only way to be sure about the presence/lack of an expander.


----------

