# vdev bad label



## levinas69 (Sep 12, 2012)

Hi all,

So, my ZFS raidz pool is reporting that it is has "insufficient replicas" because the raidz1 has "corrupted data."  The four drives that make up the array are online and they all pass SMART.  There are no faulted drives.

This happened after a power outage in which the server had a perfectly clean shutdown from UPS signal.  Not being a complete twit, I have a backup so no panic.  But, my OCD means that I really really want to understand what happened here.

My utterly unfounded guess is that the data on the drives is intact and fine but that the vdev label for the array is corrupt.  I am curious if there is a work around to recreate the raidz vdev and keep the data.  I can't see away forward at this point.  Suggestions would be much appreciated.

FreeBSD 8.2-STABLE with ZFS v4 and ZPOOL v.15
Configured with a four disk raidz: ada0, ada1, ada2, ada3.

After exporting the pool:

```
[root@neruda:~]# zpool import mpool
cannot import 'mpool': invalid vdev configuration
```

More generally:

```
[root@neruda:~]# zpool import
  pool: mpool
    id: 3532879862857622473
 state: UNAVAIL
action: The pool cannot be imported due to damaged devices or data.
config:

        mpool       UNAVAIL  insufficient replicas
          raidz1    UNAVAIL  corrupted data
            ada2    ONLINE
            ada3    ONLINE
            ada1    ONLINE
            ada0    ONLINE
```

From /var/log/messages:

```
neruda root: ZFS: vdev failure, zpool=mpool type=vdev.bad_label
```



I understand, perhaps incorrectly, that this kind of trouble can result from a bad ZFS disk label. If this were the trouble, then on one of the disks surely we would expect to see the "failed to unpack label" message as part of the output from *zdb -l* for one or more of the drives.  But, as near as I can tell, all of my labels are AOK.  Output of *zdb -l* for each of the four drives follows in the next post.

Any suggestions?

Thanks!
lev.


----------



## levinas69 (Sep 12, 2012)

*Label for ada0*

Label for /dev/ada0

```
[root@neruda:~]# zdb -l /dev/ada0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=6890223723563491244
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 1
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=6890223723563491244
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 2
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=6890223723563491244
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 3
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=6890223723563491244
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
```


----------



## levinas69 (Sep 12, 2012)

*Label for ada1*

Label for /dev/ada1

```
[root@neruda:~]# zdb -l /dev/ada1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=5843277831085397608
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 1
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=5843277831085397608
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 2
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=5843277831085397608
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 3
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=5843277831085397608
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
```


----------



## levinas69 (Sep 12, 2012)

*Label for ada2*

Label for /dev/ada2

```
[root@neruda:~]# zdb -l /dev/ada2
--------------------------------------------
LABEL 0
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=12771088994536511043
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 1
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=12771088994536511043
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 2
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=12771088994536511043
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 3
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=12771088994536511043
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
```


----------



## levinas69 (Sep 12, 2012)

*Label for /dev/ada3*

Label for /dev/ada3

```
[root@neruda:~]# zdb -l /dev/ada3
--------------------------------------------
LABEL 0
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=14995274551641648009
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 1
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=14995274551641648009
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 2
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=14995274551641648009
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
--------------------------------------------
LABEL 3
--------------------------------------------
    version=15
    name='mpool'
    state=0
    txg=4508431
    pool_guid=3532879862857622473
    hostid=1278134251
    hostname='neruda.aporia.webhop.org'
    top_guid=10872829042977914582
    guid=14995274551641648009
    vdev_tree
        type='raidz'
        id=0
        guid=10872829042977914582
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=4000799784960
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=12771088994536511043
                path='/dev/ada3'
                whole_disk=0
                DTL=27
        children[1]
                type='disk'
                id=1
                guid=14995274551641648009
                path='/dev/ada1'
                whole_disk=0
                DTL=718
        children[2]
                type='disk'
                id=2
                guid=5843277831085397608
                path='/dev/ada2'
                whole_disk=0
                DTL=717
        children[3]
                type='disk'
                id=3
                guid=6890223723563491244
                path='/dev/ada0'
                whole_disk=0
                DTL=716
```


----------

