# Any idea what may have happened?



## rusty (Jun 14, 2010)

A couple of days ago I reinstalled, everything was running superbly so I imported a 640GB drive (ada2) and a 1TB drive (ada3).

ada2 holds my videos (~400GB of films)
ada3 is backup for films, music, etc.

As it was getting late I powered down the pc. This morning I thought I'd move a TV episode from /home to ada2 only to find it didn't exist.
Neither ada2 or ada3 are online, trying to [CMD=""]zpool import[/CMD] or [CMD=""]zpool import -f[/CMD] 
return invalid vdev specification.


```
pool: disk2
 state: UNAVAIL
status: One or more devices could not be used because the label is missing 
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        disk2       UNAVAIL      0     0     0  insufficient replicas
          ada2      UNAVAIL      0     0     0  corrupted data
```

Output of ZFS' debugger
[CMD=""]zdb -l /dev/ada2[/CMD]


```
--------------------------------------------
LABEL 0
--------------------------------------------
    version=14
    name='disk2'
    state=0
    txg=126362
    pool_guid=1634257888725216332
    hostid=1453698104
    hostname='LV-426'
    top_guid=9875894291011379256
    guid=9875894291011379256
    vdev_tree
        type='disk'
        id=0
        guid=9875894291011379256
        path='/dev/ada2'
        whole_disk=0
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=640130220032
        is_log=0
        DTL=34
--------------------------------------------
LABEL 1
--------------------------------------------
    version=14
    name='disk2'
    state=0
    txg=126362
    pool_guid=1634257888725216332
    hostid=1453698104
    hostname='LV-426'
    top_guid=9875894291011379256
    guid=9875894291011379256
    vdev_tree
        type='disk'
        id=0
        guid=9875894291011379256
        path='/dev/ada2'
        whole_disk=0
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=640130220032
        is_log=0
        DTL=34
--------------------------------------------
LABEL 2
--------------------------------------------
    version=14
    name='disk2'
    state=0
    txg=124318
    pool_guid=1634257888725216332
    hostid=1453698104
    hostname='LV-426'
    top_guid=9875894291011379256
    guid=9875894291011379256
    vdev_tree
        type='disk'
        id=0
        guid=9875894291011379256
        path='/dev/ada2'
        whole_disk=0
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=640129171456
        is_log=0
        DTL=34
--------------------------------------------
LABEL 3
--------------------------------------------
    version=14
    name='disk2'
    state=0
    txg=124318
    pool_guid=1634257888725216332
    hostid=1453698104
    hostname='LV-426'
    top_guid=9875894291011379256
    guid=9875894291011379256
    vdev_tree
        type='disk'
        id=0
        guid=9875894291011379256
        path='/dev/ada2'
        whole_disk=0
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=640129171456
        is_log=0
        DTL=34
```

Nothing I've tried has got me any closer to getting these back online, and I'm gutted that my backup drive has gone down at the same time as the other drive.
Any ideas what might have caused this? 
The machine has been as stable as one could want, when turning off it's always powered down with 
[CMD=""]/sbin/shutdown -p now[/CMD].
so I'm unsure how this has occured.

Just looking for suggestions to avoid this again, aaargh all my films...


----------



## hedwards (Jun 15, 2010)

This looks very familiar to me for some reason. Is there any reason why you tried to import them this morning? Or had you exported them last night? Because unless I'm mistaken if you try those commands and there isn't another ZFS disk out there you get an error on the second command.

Actually, one thing you could try is loading up an opensolaris livecd and see if that allows you to access the data. One thing I have noticed in the past is that a ZFS set up which won't work on Linux works on FreeBSD or Opensolaris. I'm not sure why, but I thought that I'd hopelessly corrupted one myself, but it turned out that it just wasn't working under Linux. I could access it via both FreeBSD and Opensolaris.

Another thing you can try is exporting those ZFS partitions and import them again, sometimes ZFS gets a bit confused if disks change numbering.

But in a pragmatic sense, I'd recommend you do nothing until you can get a backup image of those entire drives so that you can at least reimage them to disk if need be.


----------



## rusty (Jun 15, 2010)

Thanks for the suggestions. I tried importing purely because they hadn't come online when I booted so figured it was the easiest way to get them online to see if there was something wrong, run scrub etc.
I tried the export / import but to no avail.

I think that the way I had things setup wasn't ideal (as shown here), ada3 was generally used as an rsync'd backup of ada2 but the downside being - anything corrupted on ada2 got copied in the same state to ada3. 
If they had been part of a zfs mirror / raidz then I guess the self healing would've done its part and I could've saved myself any aggravation, oh and a new external backup drive would certainly have helped 

dd_rescue is hard at work so hopefully I'll get as much data off ada2 as possible.


----------



## hedwards (Jun 16, 2010)

Sometimes what you can also do is just set them to be online. I'm not at my FreeBSD console right now, but I believe that ZFS has a specific command to do that.


----------

