# Data Recovery Tools and Testing



## jardows (Jan 22, 2021)

I've been working on a utility script for drive maintenance, and would like to include a data recovery tool.  I've been able to locate in ports sysutils/ddrescue, sysutils/dd_rescue, and sysutils/testdisk.  

The simple question is if there are additional tools, and what your general opinion is on the different tools from your experience.

The question I'm really after - what are some good ways that I could test the various data recovery tools available to determine which ones will work best for me?  I've never intentionally made data unreadable with the intention of recovering it, so I'm not sure what are the best artificial circumstances to create.  I don't get enough real data-loss events to test those scenarios.


----------



## ralphbsz (Jan 22, 2021)

What do you want to recover from?

Hardware failures? Head crashes and bad spindle bearings? Electronics board on disk drive dead? For those, you don't need software tools, you need hardware tools, such as replacement boards, STMs to scan the surface, replacement bearings or heads. All this can be done, but usually requires cooperation from the drive manufacturer (or largish companies that have stockpiles of spare parts). A lot of this work is right on the edge of espionage and criminal hacking.

Another interesting set of failures is internal metadata corruption in SSDs (beginning to also be a problem in disk drives, now that they have more and more metadata). Again, the techniques tend to be vendor-specific.

Software failures and data corruption: Overwrite a random block on disk, run fsck (or scan/scrub, on systems that support it, such as ZFS), see whether it finds it. This is very file system (or in general software) specific.

Tools like ddrescue all share a common problem: they try to get around what happens if the disk drive reacts badly, like hang for a long time, or throws errors. But the OS software stack is very good at "dealing" with these problems, for example retrying failing commands, or resetting the SCSI/SATA stack, or doing crazier things. The problem is that one typically runs tools like ddrescue on a "known bad" disk, and there errors or commands that don't finish need to be quickly reported to the tool, rather than the OS trying very hard to mask them.


----------



## Speedy (Jan 23, 2021)

In my experience when a hard drive fails then you better replace it, restore from backup and forget it. No software tool is going to get your data back after "click of death".


----------



## Mjölnir (Jan 23, 2021)

It's in base FreeBSD (/sbin): `apropos -s 1 recover` RTFM recoverdisk(1)


----------



## jardows (Jan 25, 2021)

mjollnir said:


> It's in base FreeBSD (/sbin): `apropos -s 1 recover` RTFM recoverdisk(1)


I do like options that are in base.  Have you used this for any data recovery?  I'd love to get some input on how well it works in various cases.


Speedy said:


> In my experience when a hard drive fails then you better replace it, restore from backup and forget it. No software tool is going to get your data back after "click of death".


In my job function at a large university, we are to give "best effort" to our customers.  Sometimes that is a student who just lost their thesis, sometimes it is an executive vice president who has 20 years of life work stored on a single laptop.  Or even worse, it could be the team data from one of the NCAA sports programs!  Despite us having a contract with Dropbox that provides unlimited storage for all faculty and staff, getting people to use it is, shall we say, challenging.  So while the need to do data recovery is not all that common, it is something we occasionally have to do.  Currently, we have been using Photorec/Testdisk, but as I build a boot utility for drive maintenance (I've already gotten SMART tests and data wiping figured out, writing a shell script menu system to make it easy for inexperienced technicians to use) I want to explore different options.  To do so, I need to know what would be the most effective tools for our purposes, and I want a way to objectively test the different tools and methods available.  

While we do have data recovery tasks sometimes from failing hard drives, we also get tasks like one I am currently working on, where the partition data is somehow completely gone on an external drive. This probably happened as the drive was formatted NTFS and the customer attempted to use it in a Mac, and probably "initialized" the drives or something, not realizing what they were doing. Now it is up to me to try to recover data, so I'm currently running Testdisk/Photorec on it. If the data is unrecoverable, I just tell the customer "too bad so sad" (but in a much more diplomatic way). I do not want to risk any chance of data recovery by experimenting on this drive, so thus my initial question - what would be some good practices to simulate data loss to test the various data recovery tools?


----------



## SirDice (Jan 25, 2021)

For recovery (and forensic search) there's sysutils/sleuthkit. Not easy to use though. You really need to know what you're doing to make sense of it.



jardows said:


> I do not want to risk any chance of data recovery by experimenting on this drive, so thus my initial question - what would be some good practices to simulate data loss to test the various data recovery tools?


The example you gave is easy to reproduce. Get an image of a disk with a good partition table on it and some data. Remove the partition table, then see if you can restore it without using a backup of that partition table.


----------



## Mjölnir (Jan 25, 2021)

jardows said:


> I do like options that are in base.  Have you used this for any data recovery?  I'd love to get some input on how well it works in various cases.


Years ago... IIRC I used it on a broken DVD reader?  I'm sorry I can not remember neither the results, nor the user experience.  Any serious data recovery tool should either run in readonly mode, or offer a switch to do that.  recoverdisk(1) does the former, it _copies_ data off the bad device.
On that NTFS/Mac disaster disk: I would write an e-mail to the macoOS devs (or post respective forums) and kindly ask if their partitioning tool writes a backup of the previous partition table somewhere.
In general, you can always copy the _raw_ disk (dd(1), i.e. including all blocks that the OS usually hides), then create a md(4) device backed by that file (mdconfig(8)) and use your data recovery tool on that fake device.


----------



## Crivens (Jan 25, 2021)

... and do you really want inexperienced techs around such tasks?


----------



## jardows (Jan 25, 2021)

Crivens said:


> ... and do you really want inexperienced techs around such tasks?


Teaching opportunities - this is a university after all!  Slightly more seriously, most of our technicians are Windows or Mac guys, so having a menu to help them along the way is pretty much required to see any sort of wide-spread use.  Plus I'm having loads of fun learning the shell scripting required to build such a menu!


----------



## Mjölnir (Jan 25, 2021)

jardows said:


> Teaching opportunities - this is a university after all!  Slightly more seriously, most of our technicians are Windows or Mac guys, so having a menu to help them along the way is pretty much required to see any sort of wide-spread use.  Plus I'm having loads of fun learning the shell scripting required to build such a menu!


Python or LUA are much better suited IMHO.  If your small shell script grows, and eventually it will, you're better off by having started with a more modern language; at least it should support modules.  Also QML + PyQt comes to my mind.


----------



## Crivens (Jan 25, 2021)

jardows said:


> Teaching opportunities - this is a university after all!


There is a dialog in Deadpool:
"Benefits? Like the house blowing up every 2 years?"
"Please! House blowing up builds character."

Just thinking...


----------

