# zpool import/export: hot spare unavailable



## Pfarthing6 (Jul 1, 2011)

I'm running into an odd thing. Using HAST between two nodes, works fine. I export my zpool on node1, switch HAST roles so that node2 is the primary, then import the zpool. When I check my zpool status, it shows that the hot spare is unavailable.

First Node, current master:

```
nas1# zpool add zroot spare /dev/hast/ada12
nas1# zpool status
  pool: zroot
 state: ONLINE
 scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            hast/ada0   ONLINE       0     0     0
            hast/ada1   ONLINE       0     0     0
            hast/ada2   ONLINE       0     0     0
            hast/ada3   ONLINE       0     0     0
          raidz1-1      ONLINE       0     0     0
            hast/ada4   ONLINE       0     0     0
            hast/ada5   ONLINE       0     0     0
            hast/ada6   ONLINE       0     0     0
            hast/ada7   ONLINE       0     0     0
          raidz1-2      ONLINE       0     0     0
            hast/ada8   ONLINE       0     0     0
            hast/ada9   ONLINE       0     0     0
            hast/ada10  ONLINE       0     0     0
            hast/ada11  ONLINE       0     0     0
        spares
          hast/ada12    AVAIL

errors: No known data errors
```

Export zpool on first node and switch role to secondary:

```
nas1# zpool export -f zroot
nas1# zpool status
no pools available
nas1# hastctl role secondary all
```

Switch role to primary on second node and import:

```
nas2# hastctl role primary all
nas2# zpool import -fa
nas2# zpool status
  pool: zroot
 state: ONLINE
 scan: none requested
config:

        NAME              STATE     READ WRITE CKSUM
        zroot             ONLINE       0     0     0
          raidz1-0        ONLINE       0     0     0
            hast/ada0     ONLINE       0     0     0
            hast/ada1     ONLINE       0     0     0
            hast/ada2     ONLINE       0     0     0
            hast/ada3     ONLINE       0     0     0
          raidz1-1        ONLINE       0     0     0
            hast/ada4     ONLINE       0     0     0
            hast/ada5     ONLINE       0     0     0
            hast/ada6     ONLINE       0     0     0
            hast/ada7     ONLINE       0     0     0
          raidz1-2        ONLINE       0     0     0
            hast/ada8     ONLINE       0     0     0
            hast/ada9     ONLINE       0     0     0
            hast/ada10    ONLINE       0     0     0
            hast/ada11    ONLINE       0     0     0
        spares
          dsk/hast/ada12  UNAVAIL   cannot open

errors: No known data errors
```

As you can see, the ada12 disk is now marked as "UNAVAIL" on the second node.

Does ada12 exist?

```
nas2# hastctl status ada12
ada12:
  role: primary
  provname: ada12
  localpath: /dev/ada12
  extentsize: 2097152 (2.0MB)
  keepdirty: 64
  remoteaddr: tcp4://10.10.10.13
  replication: fullsync
  status: complete
  dirty: 0 (0B)

nas2# ls /dev/hast/ada12
/dev/hast/ada12
```

...yes it does!

And to get things straightend out, I then have to remove the UNAVAIL spare and add it back again. Weird?

```
nas2# zpool remove zroot dsk/hast/ada12
nas2# zpool status
  pool: zroot
 state: ONLINE
 scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            hast/ada0   ONLINE       0     0     0
            hast/ada1   ONLINE       0     0     0
            hast/ada2   ONLINE       0     0     0
            hast/ada3   ONLINE       0     0     0
          raidz1-1      ONLINE       0     0     0
            hast/ada4   ONLINE       0     0     0
            hast/ada5   ONLINE       0     0     0
            hast/ada6   ONLINE       0     0     0
            hast/ada7   ONLINE       0     0     0
          raidz1-2      ONLINE       0     0     0
            hast/ada8   ONLINE       0     0     0
            hast/ada9   ONLINE       0     0     0
            hast/ada10  ONLINE       0     0     0
            hast/ada11  ONLINE       0     0     0

errors: No known data errors

nas2# zpool add zroot spare /dev/hast/ada12
nas2# zpool status
  pool: zroot
 state: ONLINE
 scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            hast/ada0   ONLINE       0     0     0
            hast/ada1   ONLINE       0     0     0
            hast/ada2   ONLINE       0     0     0
            hast/ada3   ONLINE       0     0     0
          raidz1-1      ONLINE       0     0     0
            hast/ada4   ONLINE       0     0     0
            hast/ada5   ONLINE       0     0     0
            hast/ada6   ONLINE       0     0     0
            hast/ada7   ONLINE       0     0     0
          raidz1-2      ONLINE       0     0     0
            hast/ada8   ONLINE       0     0     0
            hast/ada9   ONLINE       0     0     0
            hast/ada10  ONLINE       0     0     0
            hast/ada11  ONLINE       0     0     0
        spares
          hast/ada12    AVAIL

errors: No known data errors
```

I figure I can script this anomaly, but I'm not sure how that might affect a zpool that is actually using the spare when it fails over ...though I probably should test that before role out.

Oh, and yes, if I switch back to the first node again, the spare is still unavailable.

Actually, this happens even without the switchover to the other node. Just import and export the zpool on the same system and the hot spare will show as UNAVAIL.

Speaking of which, even when the hot spare is seen and a drive fails, it doesn't engage. I've added that issue to my other thread.

Am I experiencing some limitations with the FreeBSD implemenation of ZFS or am I totally missing something?

Any helpful hints would be greatly appreciated!

thanks!


----------

