# Boot from SAN and Multipath



## serveroneblu (Nov 13, 2014)

Hello,

Our organization has acquired HP DL380 with one Dual Port HBA Qlogic Qlogic2562 8GB (HPAJ764A)

The server has been connected to the Storage Area Network (3Par) and I have installed FreeBSD 10.0 STABLE (amd64) on a LUN with the ZFS filesystem. I have used ZFS partitions with Pool Name: BootSAN and GPT schema on da1 so *I* get:

```
Filesystem              Size    Used   Avail Capacity  Mounted on
BootSAN/ROOT/default     45G    1.2G     44G     3%    /
devfs                   1.0K    1.0K      0B   100%    /dev
BootSAN/tmp              44G    176K     44G     0%    /tmp
BootSAN/usr/home         44G    144K     44G     0%    /usr/home
BootSAN/usr/ports        45G    828M     44G     2%    /usr/ports
BootSAN/usr/src          45G    545M     44G     1%    /usr/src
BootSAN/var              44G    220M     44G     0%    /var
BootSAN/var/crash        44G    148K     44G     0%    /var/crash
BootSAN/var/log          44G    328K     44G     0%    /var/log
BootSAN/var/mail         44G    148K     44G     0%    /var/mail
BootSAN/var/tmp          44G    200K     44G     0%    /var/tmp
```

The boot from SAN is working perfectly. I am able to see all four paths for the LUN 0:

```
camcontrol devlist |grep VV
<3PARdata VV 3210>                 at scbus3 target 0 lun 0 (pass1,da1)
<3PARdata VV 3210>                 at scbus3 target 1 lun 0 (pass3,da2)
<3PARdata VV 3210>                 at scbus4 target 0 lun 0 (pass5,da3)
<3PARdata VV 3210>                 at scbus4 target 1 lun 0 (pass7,da4)
```

`gpart` output is:

```
# gpart show
=>       34  104857533  da1  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da2  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da3  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)

=>       34  104857533  da4  GPT  (50G)
         34       1024    1  freebsd-boot  (512K)
       1058    4194304    2  freebsd-swap  (2.0G)
    4195362  100662205    3  freebsd-zfs  (48G)
```

I have activated geom_multipath and Qlogic driver directly at boot*:*

```
cat /boot/loader.conf
geom_multipath_load="YES"
isp_load="YES"
```

But when *I* try to enable the multipath:

```
sysctl kern.geom.debugflags=0x10
gmultipath label -v BOOT /dev/da1 /dev/da2 /dev/da3 /dev/da4
```

In `dmesg` *I* see*:*

```
GEOM: multipath/BOOT: corrupt or invalid GPT detected.
GEOM: multipath/BOOT: GPT rejected -- may not be recoverable.
GEOM: diskid/MULTIPATH-%201653047: corrupt or invalid GPT detected.
GEOM: diskid/MULTIPATH-%201653047: GPT rejected -- may not be recoverable.
```


```
gmultipath status
          Name   Status  Components
multipath/BOOT  OPTIMAL  da2 (ACTIVE)
                         da3 (PASSIVE)
                         da1 (PASSIVE)
                         da2 (PASSIVE)

gpart status
                        Name  Status  Components
                       da1p1      OK  da1
                       da1p2      OK  da1
                       da1p3      OK  da1
             multipath/BOOTa      OK  multipath/BOOT
```

First problem, the configuration looks OK but when I detach a fibre, the OS will hang. Second, after a reboot the OS won't boot, *I* have this message on the console:

```
FreeBSD/x86 boot
Default 0:ad(0,a)/boot/kernel/kernel
Boot:

The system cannot find the boot
```

I tried some commands, but I'm failing miserably 

`- echo "0:da(1,a)/boot/loader" > /boot.config`
`- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1`

In /boot/loader.conf:

```
vfs.root.mountfrom="zfs:/BOOT"
```

My question is*:* is it possible to have write access on a FS like / at boot via the geom_multipath device?

Thank you for your help.


----------



## usdmatt (Nov 13, 2014)

I think there are a few problems here (although I don't have any experience with multipath and booting from fibre-channel)

It appears that gmultipath writes metadata to the end of the disk. This is probably colliding with the backup GPT table. I'm not sure of the "correct" way to deal with this. The gmultipath man page doesn't mention anything about using it in conjunction with GPT. Maybe you could pass the GPT partition to the create command instead, so that gmultipath metadata is stored at the end of the partition and not at the end of the disk?
`gmultipath label -v BOOT da1p2 da2p2 etc`
The pool will need to be told to use the multipath device. If you created the pool before multipath, it will be using something like /dev/da1p2 (You can see this info in `zpool status` output). If that one device disappears (which it will if the relevant path disappears), the pool and system will stop working. The pool needs to use the /dev/multipath/BOOT device, which is the one that will use any of the available paths. I'm not sure how you can change this at this point without a bit of messing about. It would of been easiest to create the multipath device, then use that to create the pool during installation.
The entry in /boot/loader.conf should look something like the below (although if it's booting, I suspect you have a new enough version of FreeBSD that it is using the bootfs property rather than relying on this loader variable).

```
vfs.root.mountfrom="zfs:BootSAN/ROOT/default"
```


----------



## serveroneblu (Nov 13, 2014)

Ciao usdmatt!
Thank you very much. Using this command on the boot partition the GPT was ok without errors: `# gmultipath label -v BOOT /dev/da1p1 /dev/da2p1 /dev/da3p1 /dev/da4p1`. Now I would like to boot with the new label "BOOT", so *I* tried this command: 
`# disklabel -Brw /dev/multipath/BOOT auto`
`# mount -o rw /dev/multipath/BOOT /`

I tried also with this entry in /boot/loader.conf:

```
vfs.root.mountfrom="zfs:/dev/multipath/BOOT"
```
but it does not work.

How can I use the new label "BOOT" for / at boot via the geom_multipath(8) device?

Is _it_ possible to change the pool from "BootSAN" to "BOOT"?


```
# zpool status output:
  pool: BootSAN
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        BootSAN                                       ONLINE       0     0     0
          gptid/40ca71b7-6040-11e4-89c5-38eaa715862c  ONLINE       0     0     0
```


----------

