# Problem replacing drive in ZFS array



## garrettmoore (Jul 17, 2010)

I have an 8-drive ZFS array consisting of WD15EADS drives. One of my disks has started to fail, so I got a replacement disk. I have replaced a disk before by:

`zpool offline tank /dev/da5`
shutting down, swapping from old disk to new disk
booting
`zpool replace tank /dev/da5`

This worked fine.

This time the failing disk was da3, and I tried the same thing:
`zpool offline tank /dev/da3`
zpool status showed da3 offline.
shut down, swapped old disk to new disk.

When I booted again, I got:

```
NAME        STATE     READ WRITE CKSUM
        tank        UNAVAIL      0     0     0  insufficient replicas
          raidz1    UNAVAIL      0     0     0  corrupted data
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
```

I switched back to the old disk and booted again and then I could access my data again, and da3 still showed as offline. I tried 'zpool online tank /dev/da3' and after a few seconds resilvering completed and all 8 drives are back online again, but with the 'dying' disk as da3 still.

I tried shutting down WITHOUT first offlining /dev/da3, and swapping the disks, and when I booted I again got 'insufficient replicas'.

Why am I getting this error, and how come it worked ok the last time I replaced a disk?

Thanks.


----------



## olav (Jul 19, 2010)

Have you ever "scrubbed" the pool?


----------



## garrettmoore (Jul 21, 2010)

Turns out to have been a problem with the disk controller re-ordering the disks when I swapped a disk.

Instead of shutting down to switch disks, I hotswapped them, using camcontrol.

`# zpool offline tank da3`
`# camcontrol stop da3`
swap drive
`# camcontrol rescan all`
*"da3 lost device, removing device entry"*
`# camcontrol rescan all`
*"da3 at mpt0 ..."*
`# zpool replace tank da3`

Done. (actually I have some other issues with the replacement disk but those are unrelated to this thread)


----------

