# Restoring deleted file from ZFS file system



## churchi (Jul 8, 2010)

Hi All,

This morning I have accidentally deleted some files in a certain directory on my ZFS storage system. I am wondering if there is a way to get these back? I didn't have an off line backup of these files, so I am wondering if I can use some file system restore tools?

I am not sure if I had auto snapshots setup, however is there a way to check and if i didn't have snapshots setup can i recover the files from the ZFS file system?

Thanks.


----------



## phoenix (Jul 8, 2010)

Check the output of `$ zfs list -t snapshot poolname/filesystem` to see if there are any snapshots of the filesystem.  If there are, you can copy the file from /path/to/filesystem/*.zfs/snapshot*/snapshotname/path/to/file/  (replace /path/to/filesystem and /path/to/file).

If you do not have snapshots, the file is gone.


----------



## churchi (Jul 9, 2010)

Thanks Freddie,

Here is the output:

```
[root@server-01]# zfs list -t snapshot
no datasets available
[root@server-01]#
```

I guess I am out of luck as I had no snapshots setup on my system.


----------



## S3TH76 (Mar 21, 2016)

Is there any other ways to recover this file? I'm interested because I placed myself in a similar position.


----------



## SirDice (Mar 21, 2016)

And that is why you should have backups.


----------



## S3TH76 (Mar 23, 2016)

SirDice - well, it was an accident. We are 2 users with rwx on a /www/test.site.com  directory and when I was transferring a 3 Gb ISO file at half of it transferr was stopped and I saw that entire directory structure of test.site.com including the /tmp directory where I was copying the file was deleted by my coleague to upload the site with an updated platform of it. But I lost the *.filepart of 1.5Gb... This was a ocassionally file transferr so no backup was made to it.

In other order of ideeas: in Windows OS (any) you have many free or paid software for recovery of data, under Linux/Unix-like OS I don't know or hear/saw anything about it.


----------



## usdmatt (Mar 23, 2016)

Seems incredibly strange to me to resort to looking up recovery solutions just for a file transfer that didn't complete...

Regarding accidentally deleted files, snapshots are one of the primary features of ZFS, and if you're running ZFS, you may as well take regular snapshots even if you don't really have a use for them. At the least it allows you to go back and retrieve removed files, or something you've managed to mess-up/overwrite (assuming the file existed when a snapshot was taken). 

Also, there is no auto-snapshoting built in. ZFS or FreeBSD have no interest in managing snapshots for you. You either have to take snapshots manually, write your own cron script, or use one of the many ports.


----------



## SirDice (Mar 23, 2016)

S3TH76 said:


> In other order of ideeas: in Windows OS (any) you have many free or paid software for recovery of data, under Linux/Unix-like OS I don't know or hear/saw anything about it.


Because, generally speaking, the Linux/Unix-like systems are administered by people that actually understand the significance of proper backups. So there's very little need for it. 

That said, there is some recovery software like sysutils/autopsy/sysutils/sleuthkit. But as far as I know it doesn't support ZFS. In any case it's going to be a long and tedious exercise with zero guarantees you'll be able to recover anything. Having to spend several days on analyzing is the prime reason why everybody else simply restores from backup, as that usually only takes a few minutes.

https://www.joyent.com/blog/zfs-forensics-recovering-files-from-a-destroyed-zpool


----------



## tingo (Mar 23, 2016)

Real operating systems don't have training wheels. You fall off, you get back on again and (hopefully) you learn not to make the same mistakes again.


----------



## David Schwartz (Apr 8, 2016)

tingo said:


> Real operating systems don't have training wheels. You fall off, you get back on again and (hopefully) you learn not to make the same mistakes again.


It really boggles the mind that anyone can know enough about computers to go through the trouble of installing ZFS (or an entire OS that comes with it), and not know enough to back up their data! How exactly does that work?

To be perfectly honest, the number of people asking about "data recovery" online is really frustrating. They screw up, and then it becomes everyone else's problem to get their data back.


----------



## sfm1 (Feb 13, 2018)

"...and not know enough to back up their data! How exactly does that work?"

Because things still happen some times, that is how.
I am a life-time server operator and have at times lost files, and googled for if there is a way to get it back.

Does somebody claim that they could not delete files by mistake and thereby having lost them?

"They screw up, and then it becomes everyone else's problem to get their data back."
So, if I have a problem and ask if there is a way, then it 'it becomes everyone else's problem'?

I can't believe the unfriendlyness in this forum.
But thanks to phoenix, usdmatt and after all SirDice who wised up and gave a helpful reply.


----------



## ShelLuser (Feb 13, 2018)

First of all please keep in mind that you're responding to a thread which is nearly one year old. Some of the messages you responded to are even older, so there's really not much sense to it because it remains to be seen if those users are still active.

Also: if you want to comment on someone it helps to use the quote feature (or multi quote if you want to reply to multiple messages). This way the users will actually get an alert that someone commented (which increases the chance of getting a response).



sfm1 said:


> Does somebody claim that they could not delete files by mistake and thereby having lost them?


That's really not the point. The point is mostly about people not thinking ahead. A filesystem like ZFS makes it really easy to avoid problems like these, but if people don't use them then yeah... You can't really "blame" the filesystem or its features for that, which is what most people tend to do.



sfm1 said:


> I can't believe the unfriendlyness in this forum.


Being direct is not the same as being unfriendly. This thread didn't even come close in my opinion. With technical stuff it's even preferred to be direct in order to avoid the possibility of creating extra confusion.

There's also something as perception and intent. Just because someone is being direct doesn't mean he intended to be unfriendly. Instead of immediately assuming the worst you could also pick it up for what it was: a mere comment on a forum.

There's also something as being over sensitive and incapable to handle criticism. If that applies then, sorry to say, but the technical Internet as a whole (referring to technical fora) are probably not the best places to roam. Because people tend to be direct for reasons mentioned above.


----------



## lefsha (Mar 1, 2018)

ShelLuser said:


> That's really not the point. The point is mostly about people not thinking ahead. A filesystem like ZFS makes it really easy to avoid problems like these, but if people don't use them then yeah... You can't really "blame" the filesystem or its features for that, which is what most people tend to do.



As for me, you can be as much direct as you wish, but the point is you are wrong! in your way of thinking!

NOBODY!!! can think ahead of situation which is just happened. In my >20 years experience with *nix I've never had an issue with
files I deleted years ago. Even I would fall into it I would probably never remember, that something has been deleted.

In 99.99% of cases you just delete files without intention and that moment you recognize that this has happened.
Also usually you did something with that file or group of files at that very last moment, before deleting them.

For example it just has happened to me now. I never meant delete my PG config files instead I did this:

```
pg_dropcluster --stop 9.1 main
```

Now all my config files from.... /etc/posgressql just gone. To me it's a bug in software it cannot delete any config files without implicit confirmation from user. But now it became ZFS issue.

If I cannot restore all these files which is very easy on very old FAT, whom I have to blame?

Your position to blame me for not doing a backup. Are you personally doing backups every 2 minutes??? - Probably not.

Therefore you are totally wrong! Or better to say you have no clue. I am not too direct?

So I suggest you don't talk to people like that. You aren't more clever than others.  Therefore you either help or stay calm = shut up.

P.S. If all those who blamed people for not doing backups would know just a little how really almost all FS work, they would fully understand
the wish to undelete just deleted file. Almost none of them really delete files. Therefore FAT and EXT files systems do have undelete feauture as long as you don't use freed blocks again. And that IS the case if you just deleted a file.


----------



## Snurg (Mar 1, 2018)

Well, I find it sad that HAMMER is available only on Dragonfly BSD 
With HAMMER you just could look up the version history of the file, and take the one you like best...

With a cronjob doing regular ZFS snapshots one can approach that HAMMER functionality somewhat, though.


----------



## lefsha (Mar 1, 2018)

Not only HAMMER. Absolutely EVERY file system has this capability. Only encrypted file systems with additional security do really delete files.
And that it a nightmare in terms of performance. Nobody did it for ZFS, just because of ignorance.
I am not happy, but can live without it. Have now to spend another hour or more to recover text file I deleted.

I just cannot tolerate answers of idiots who think of themselves like they are guru and capable to teach people.
Instead they just have no clue!

The proper answer in such a case would be -"Hey, sorry for what has happened, but ZFS has no such utility to recover your data."
Can be used as template I provide it on BSD License

P.S. Also sorry for my style. I was pissed off by ZFS. Have installed it specially on a MIRROR to improve storage safety. Have spent several days
to make it right. And have got a surprize, which I would never see on my EXT4 I have everywhere else. Argh..


----------



## Snurg (Mar 1, 2018)

Didn't know that restoring of deleted files would be easy on ext4 or any unix filesystem... thought deleted=gone.
Thought that this would be one of the things HAMMER introduced... one can view it as sort of filesystem version control system, even as forensic tool to view what malware on VMs changes, for example


----------



## gofer_touch (Mar 1, 2018)

Yeah, it should be possible to get approximate functionality from ZFS by careful snapshot and prune policy. With that being said I think it would be extremely beneficial having HAMMER in FreeBSD as it has some tools which are simply better for this kind of thing. Not to mention a better implementaton of deduplication.


----------



## ShelLuser (Mar 1, 2018)

Well, this thread seems to be getting more hilarious by the day. But sorry to say but the days of MS-DOS' FAT16 which provided a pretty static File Allocation Table (which made it practically impossible not to spot the location where missing files could have gone) is long after us.

Yet even with FAT16/FAT32 it's pretty easy to lose your files, for the simple reason that once the FAT gets updated then best of luck to you finding all the bits and pieces of your data, especially when they're scattered on a fragmented filesystem.

Hmm, what was that saying again? Talking without being hindered by any factual knowledge? Something like that 

Bottom line: think ahead, make backups, don't blame the system for your own short comings. Unless of course you're working for GitLab, in which case I guess it's part of your job.


----------



## lefsha (Mar 1, 2018)

It seems that some people without a clue in everything are most aggressive to teach others! They just can't stop do it even
after they were proven of being wrong!

Please, answer messages only if you wish to HELP! not to blame other people. You definitely do not know their situation
and can not charge on it. To do that one need to have certain level of qualification in the topic. Obviously it's absent in your case.

It seems FBSD has a real problem to survive in that world. Linux users seems to be much more educated what concerns computers, OS and FS.

Just links: https://wiki.archlinux.org/index.php/file_recovery https://www.cgsecurity.org/wiki/Advanced_FAT_Repair
Key words: extundelete, testdisk

Such tools are existed for almost every FS, if it has a certain level of popularity. They all are based on simple principle - none of FS do really delete files!
If immediately after deleting a file one would try to recover it the success probability is practically 100%.

Almost anyone who were using Linux more than a year usually know about them. They work perfectly well!
Both helped me to save many files over long time.

P.S. Now I am ready to listen either a prove, that I am wrong, or an excuse about being not well informed.


----------



## lefsha (Mar 1, 2018)

Snurg said:


> Didn't know that restoring of deleted files would be easy on ext4 or any unix filesystem... thought deleted=gone.
> Thought that this would be one of the things HAMMER introduced... one can view it as sort of filesystem version control system, even as forensic tool to view what malware on VMs changes, for example



Snurg, please, see my previous message in that thread. I have provided links for programs you can use.
Such tools are existed since 90x! Possibly even earlier. I had no PC in 80x, therefore do not know.

The tool "extundelete" has it's name based on DOS tool "undelete" which was very poplular many years ago.

Unfortunately, those local experts seems to be totally unqualified, so if you wish to learn something real, try to avoid them. Good luck.


----------



## gkontos (Mar 1, 2018)

lefsha said:


> It seems FBSD has a real problem to survive in that world. Linux users seems to be much more educated what concerns computers, OS and FS.



ZFS is irrelevant from FreeBSD. ZFS was ported to FreeBSD and is currently being also used in Linux too.
Perhaps it would be better for you to just use ZFS on Linux.


----------



## SirDice (Mar 1, 2018)

lefsha said:


> Such tools are existed for almost every FS, if it has a certain level of popularity. They all are based on simple principle - none of FS do really delete files!
> If immediately after deleting a file one would try to recover it the success probability is practically 100%.


Have you actually tried using any of them in a real world situation? I have. From my own experience I can tell you the success rate is far less, and is probably closer to 50/50.

In a real world situation you can't just turn off a production system because some id-10-t removed a file they shouldn't have. And just the act of shutting down a system could cause writes (caches being flushed to disk) that could overwrite the old file data. Yeah, you can risk it and forcefully power down the machine. But that could result in even bigger problems (databases for example) than just a lost file.

The advice is simple, don't rely on magical tools. Rely on your backups. Because at the end of the day, that's going to save you.


----------



## ralphbsz (Mar 1, 2018)

Hammer is by far not the first file system with snapshots.  It's snapshot mechanism is more fine-grained than common, but it pays a price for that in that delete operations are handled badly (slowly, inefficiently).  The other well-known file system with fine-grained snapshots is WAFL (which is only available inside NetApp filers, meaning NFS appliances); WAFL is so well-engineered that it gets very good performance and a complete CoW (copy-on-write) deisgn.  Yet it does not have undelete support.  The most extreme example of a file system with snapshots was "Elephant - the file system that never forgets" (look up the research papers about it).  Alas, Elephant was never made into a full production file system that shipped with any operating system, and remained a (glorious) research project.

All engineering is an art of making compromises.  Implementing a file system is a compromise between performance, durability, and features.  Snapshots are one ingredient in such a compromise.  It is possible to make extremely fine-grained snapshots, such as in Elephant, but the result is not production worthy.  As SirDice said: If you want protection against unintentional delete, use backup and snapshots.

Tools that try to undelete files after the fact are voodoo.  They may work some of the time, but they are just unsafe.  If they could be implemented efficiently and safely, they would be, but the fact that file system developers do not release those tools to the public should tell you that they should not be used.  With expert help, it is sometimes possible to retrieve files that have been lost and verify their correctness.  I've had to do that for file systems I worked on, but I would refuse unless I get a large paycheck, and there is a strong indemnification agreement that makes sure I'm not liable if something goes wrong.

A poster above claimed that Linux is better than FreeBSD because it has such tools.  That poster is so wrong that it would be better for FreeBSD if he goes back to using and hyping Linux.


----------



## ShelLuser (Mar 1, 2018)

I'm also wondering why this 8 year old thread needs to be hijacked anyway. Its obvious that the current topic is totally unrelated to that of the OP because many things changed in 8 years time. Instead of hijacking an existing thread and making a nuisance of yourself (which, on a personal impression, makes you really stand out as a troll) it might be a better to start a new one to fully explain the current issue at hand.

Provided this is actually related to FreeBSD of course, something which I now seriously question.


----------



## lefsha (Mar 1, 2018)

SirDice said:


> The advice is simple, don't rely on magical tools. Rely on your backups. Because at the end of the day, that's going to save you.



1. The main point of discussion was how to properly answer to questions people are asking.
No backup, no preparation could do anything about what people are asking. None of us is able to predict the future and what would happen next.

And I insist it's a bad attitude trying to teach people, who were not asking any one to teach them!!!
Is it so difficult to understand? Reading this forum or Stack Exchange/Overflow I am always wondering where those people come from?

Somebody is asking "how to perform a certain task" and very common answer is "You don't need it" or "Why do you need it?"
or "It's a bad idea - never do it"  etc.

Who ever told you, that you do have enough expertise to give such suggestions? In many cases the first answer is something like it doesn't work and the second one somebody who really knows provide the solution.

Why it's so difficult to keep quite if you don't know the answer? Or at least you are willing to say something nobody is asking about???

Imagine some one would ask you what is better - doing backups or use an "undelete" program for files recovery.
Then it would be OK. You provide your opinion. Although we all know those questions are not allowed at SO/SE... Think why it is so?
Because too many so called experts are trying to convince people in some stupid things.

2. Regarding backups! That was not a topic! Nobody is denying the need of backups, but they aren't useful in many cases!
It's not without a reason that Microsoft has introduced the "Recycle Bin" technology which is mad simple to prevent loosing files in exactly
such circumstances. And now after 20+ years I am reading that is called "a Magic Tool"...

Do you really wish to say, that Windows 3.11 is more advanced than today's FreeBSD?????? - OMG...


----------



## lefsha (Mar 1, 2018)

gkontos said:


> ZFS is irrelevant from FreeBSD. ZFS was ported to FreeBSD and is currently being also used in Linux too. Perhaps it would be better for you to just use ZFS on Linux.



Hm... May be we read different words and different text, but for me that topic is irrelevant to: FreeBSD, Linux, ZFS etc...

Thanks for suggestion. I am currently doing exactly that running ZFS on Debian. After playing 3 weeks with FBSD I could I see that OS remained in the past. I could solve some issues I had, but on a long run it is not feasible to keep it running without spending much time for problem solving.
I have mentioned in that forum (different thread) about OS reboot without a reason. Which is not happening now on Debian. Hardware is obviously the same. I could not blame FBSD developers team, because it's too much work to address all those issue. And unfortunately the installation base does matter. I did experiment with piping "yes" command. FBSD version gave me 30 Mb/s, Nim on FBSD gave me 1Gb/s the same was with updated C version. Debian gave me 8 Gb/s on the same system... I would _not_ write about it, if I would not get your suggestion...

My suggestion is to think about your words, before typing them. 

If in addition to that: _if_ the community is unfriendly it's just a matter of time when FBSD will disappear. Do I wish that? - Not at all.
I personally found this community very friendly excluding some people I mentioned above.


----------



## lefsha (Mar 1, 2018)

OMG... How many experts like that are using FBSD?



ralphbsz said:


> Tools that try to undelete files after the fact are voodoo.



For my record... A damn simple "Recycle Bin" in Windows is a voodoo...
Do you need some help to buy you a book about file systems?



ralphbsz said:


> A poster above claimed that Linux is better than FreeBSD because it has such tools.  That poster is so wrong that it would be better for FreeBSD if he goes back to using and hyping Linux.



Oh! That guy not only have no clue about FS, but he can't read either... No wonder why so difficult to understand how FSs work.

I did not mentioned about Linux is better, than FBSD!
But if you can point me to that I would pay you any book about file systems you can buy!

The fact is and that what was really mentioned. Almost every Linux user knows about undelete programs OP was asking about.
That statement has nothing!!! in common with a statement Linux is better, than FBSD. Especially there is no such OS like Linux
People who know a little about computers would never compare a whole OS with just a kernel!
Just for those who overslept last 20 years... Android use Linux as a kernel. Debian is using FBSD kernel as an option...

But despite that, every time we see people who keep comparing Apples with Cars or Linux with FBSD...

Ah those experts...  Actually I don't blame them. It's fully OK for people don't know something. Much worse is of those "experts"
are starting  to teach others...

Please, don't do it. And then you can compare Linux with your Grandmother or smoke another grass. Thank you very much!


----------



## Eric A. Borisch (Mar 1, 2018)

lefsha said:


> I did experiment with piping "yes" command. FBSD version gave me 30 Mb/s, Nim on FBSD gave me 1Gb/s the same was with updated C version. Debian gave me 8 Gb/s on the same system...



We need a new internet rule/law for any time the discussion turns to how fast `yes` is.


----------



## ShelLuser (Mar 1, 2018)

Use the best tool for the job, that's what most professionals do. Considering all the above arguments I suppose that could indeed be Windows 3.11 in this example, so that would be my advice.

There are indeed situations where one environment is better in place than the other. So I don't see how this would be any different.


----------



## lefsha (Mar 1, 2018)

ShelLuser said:


> Provided this is actually related to FreeBSD of course, something which I now seriously question.



Nope. It was purely personal. I do admit that. While looking how to recover a file I've lost under ZFS I found that post.
It has no time relation, because such problems were happening 20+ years ago and will happen 20+ years later.

The problem is not FBSD, neiter ZFS.  The problem is people like you who blame others for nothing. The whole thread would make
more use for everyone if people would just mention that up to now according to "my" knowledge nobody has written such undelete tool
specially for ZFS. That is it.

Instead you started a personal discussion about what people have to do and don't have to. Do you really think it was OK?

Here I am ready to stop it. And I repeatedly admit, that I do not blame FBSD, which was my very first OS about 20 years ago!
I do not blame ZFS and do not blame any one in this forum. I am just asking every one to behave like people and
not being aggressive specially to those who fall into a trouble! But if you behave that way you can expect the same behaviour towards your person.

I hope it's clear by now. Thank you very much and sorry if you feel bad about my words. I just want to show you how other people feel if
people behave like you did. Thx


----------



## rigoletto@ (Mar 1, 2018)

Hilarious!

Someone is arguing that other people should have created a recovering tool for ZFS, a file system which one of its main features is the ability to easy (or ever automatically) take snapshots with the possibility of remote replication, yes...

With ZFS I do not ever use a trash can on my desktop, I do not need one.


----------



## lefsha (Mar 1, 2018)

Eric A. Borisch said:


> We need a new internet rule/law for any time the discussion turns to how fast `yes` is.



It's offtopic here. The roots aren't in "yes", but in pipes... Personally I don't care how fast "yes" is.


----------



## lefsha (Mar 1, 2018)

lebarondemerde said:


> Hilarious!
> 
> Someone is arguing that other people should have created a recovering tool for ZFS, a file system which one of its main features is the ability to easy (or ever automatically) take snapshots with the possibility of remote replication, yes...
> 
> With ZFS I do not ever use a trash can on my desktop, I do not need one.



You have missed the point. Completely! Totally! Absolutely! Read it once again or just forget it.


----------



## Eric A. Borisch (Mar 1, 2018)

lefsha said:


> It's offtopic here. The roots aren't in "yes", but in pipes... Personally I don't care how fast "yes" is.



Sorry, but no. (Pun intended.)

https://forums.freebsd.org/threads/...issue-on-freebsd-in-c-code.64778/#post-378428

(TL;DR: you can make the bulk of the difference go away by making FreeBSD’s `yes` more aggressive.)

But I absolutely agree that I don’t care how fast `yes` is. It is unfortunate that some have chosen to use it as a benchmark.


----------



## k.jacker (Mar 1, 2018)

`# touch /root/bin/rm.sh`

```
#!/bin/sh
mv $@ /tmp/recycle-bin
```
`# mkdir /tmp/recycle-bin`
`# chmod u+x /root/bin/rm.sh`
`# alias rm /root/bin/rm.sh`
Now comes the accident...
`# rm /etc/rc.conf /etc/sysctl.conf`

```
# ls -l /tmp/recycle-bin
total 8
-rw-r--r--  1 root  wheel  748 23 Jan. 11:23 rc.conf
-rw-r--r--  1 root  wheel  557 30 Nov. 18:53 sysctl.conf
```
Lucky me...  `mv /tmp/recycle-bin/*.conf /etc/`

You can still `/bin/rm/[I]filename[/I]` if you really want to remove a file.
Of course it needs more tuning, like permissions, depending on your needs, adding it to path maybe and remove the options from `rm` if given any...
I actually like that, I'm going to fine tune that a bit


----------



## ralphbsz (Mar 2, 2018)

lefsha said:


> For my record... A damn simple "Recycle Bin" in Windows is a voodoo...


Sorry, but you have your facts wrong.  Indeed, both Windows and the Macintosh have recycle bins or trash cans on the GUI.  But these are not undelete functions.  Matter-of-fact, neither NTFS on Windows, nor HFS/APFS on the Macintosh have an undelete function.  Really, go to their API reference and look for them, and it is not there.  Or do the following experiment: Open a shell window on either operating system, and delete a file with the rm command (on a Mac) or the del command (on Windows).  It does now show up in the recycle bin/trash can.  Then write a tiny little C program that makes the unlink() system call, and use it to delete a file; or do the equivalent in Perl or Python or the scripting language of your choice.  None of the deleted files show up in the recycle bin, and you can't get them back.

Here's why: The delete function that's built into the graphical file manager of these operating systems actually doesn't delete files.  Instead it moves them (that is, renames them) to a special directory which is usually invisible in file system traversals, and visible instead as a recycle bin or trash can.  Restoring a file from the recycle bin is another move or rename operation.

If you had tried the "pg_dropcluster" operation you originally complained about on Windows (or on a Mac, or the equivalent operation for these OSes), you would not have found the missing files in the recycle bin, because they wouldn't have been there

Now, if you like to, you can replace the "rm" command at the shell level with a script, for example like k.jacker demonstrated above.  This is an old idea, and various versions of these scripts are floating around the internet.  They all have some sort of problem.  For example trying to move a file across mount points may be very slow, they need a mechanism to resolve name collisions (you delete a file named a.b from one directory, and then a different file called a.b from a different directory, and then what happens), and you get ownership problems with the recycle bin directory, if two different users both delete files with the same name.  But their big problem is that they only work when the deletion happens through the explicit "rm" command.  They don't help if the file is deleted by overwriting (try "cat /dev/null > /etc/passwd" sometime), or if the deletion happens from a program other than rm (for example from a script written in python or perl), or as part of an operation like the "pg_dropcluster" example you gave above.

I repeat my statement: I know of no file system that has an explicit undelete function.  If you find a counterexample, I would be interested to learn about it.  However, there are file systems that have fine-grained snapshot functions.  Everyone knows about the snapshots in NetApp filers (they really popularized them), about the ones in ZFS, and about the ones in Hammer.  Fewer people know the version support in the VMS file system, which fundamentally created a snapshot of a file everytime it was closed after modification (that's because the VMS file system has not been heavily written about in the academic literature, the only paper I know about it is Kronenberg et al., and it is sold only with an operating system that has had little market share in the last decade).  Even fewer people know about the Elephant file system, which created a snapshot after every write operation.  But in both VMS and Elephant, a file was really gone after you deleted it.  Anecdote: The main author of the Elephant file system was a colleague of mine for several years.

Would you like to try again, with different alternate facts?



> Do you need some help to buy you a book about file systems?
> Oh! That guy not only have no clue about FS, but he can't read either...


Thank you, but no thank you.  I have plenty of books, and stacks and stacks of research papers and patents about file systems.  Matter-of-fact, some of those papers and patents were written by me.



> I did not mentioned about Linux is better, than FBSD!


Quotes from you: "Therefore FAT and EXT files systems do have undelete feauture ..." and "Absolutely EVERY file system has this capability" and "Nobody did it for ZFS, just because of ignorance" and "which I would never see on my EXT4 I have everywhere else".  You seem to state that ext4 has this feature (that means on Linux, because ext4 is a native file system on Linux), and that ZFS does not.  You also repeatedly complain about other issues you have with FreeBSD and with this community, and that you are going back to using Linux.

Finally, let me summarize the discussion.  Undelete tools exist for some file systems.  They are all unreliable and dangerous, some more some less.  Under human manual supervision with an expert in file system internals at the control, you can sometimes do better than automated undelete tools, but not always.  Many modern file systems support some notion of snapshot, ZFS definitely does.  Intelligent use of snapshots removes the need for undelete tools.


----------



## Eric A. Borisch (Mar 2, 2018)

ralphbsz said:


> I know of no file system that has an explicit undelete function. If you find a counterexample, I would be interested to learn about it.



It appears (I haven't used it) that hammer has some ability to undelete: https://www.dragonflybsd.org/cgi/web-man?command=undo .. it looks to rely a nightly (cron-initiated) prune of deleted files, but there is some ability to recover until then. Certainly an interesting design, and not one I've seen before.

That said, I can understand the (recent) poster's frustration. To him, I say this: we've all been there at some point. It's hard when you're extremely frustrated to see that others are trying to help, even if it is in the form of "you should really have done this." No one (to my knowledge) has written a (publicly available) tool for undeleting zfs files. Since snapshots are a flagship feature of the system, I don't expect anyone will (well, someone who charges to help disaster recovery operations might have one, but they will likely charge for its use.) See, for example, a similar discussion here: http://list.zfsonlinux.org/pipermail/zfs-discuss/2016-March/024994.html ... if you were looking for a simple text file, you might have been able to find snippets with grep of the raw device, but for a database (presumably of some size) that is constantly being re-written in small chunks... you're looking at a very tall order. (If compression is enabled, grep would be of no use for a text file, either.)

I know it is after the fact, but I would suggest installing sysutils/zfstools and turning on auto-snapshots; this has saved my bacon more than once. Good luck!


----------



## Oko (Mar 2, 2018)

Eric A. Borisch said:


> No one (to my knowledge) has written a (publicly available) tool for undeleting zfs files.


To be frank with you from where I am sitting a properly implemented delete functionality would be much more desirable than "undelete" BS OP is asking about. Let me explain. ralphbsz will correct me but my understanding that *rm* command on the ext2 file system just removes inode numbers. I find this post semi-informative

https://unix.stackexchange.com/questions/117325/where-are-filenames-stored-on-a-filesystem

about the notions of inodes and where the file names are actually stored. In practical terms that means that contrary to ralphbsz said in his post the content of the file is indeed never completely removed unless it is overwritten. You might need a help of local NSA gurus to recover the data from such drive but that should not be a show stopper. I don't know enough about design of various file systems to tell how OP's OS of choice (Linux) picks up the blocks where the new content is written. However even writing zeros on the top of existing data is not enough to truly delete file (I think this is what Linux command *shred* does). There is a way to recover data from the drive which is overwritten in such linear fashion with non-random stream of zeros and NSA guys are doing it routinely. I am not sure how FreeBSD is handling *rm* command and how ZFS affects it but OpenBSD *rm* actually rewrites blocks three times with the pseudo-random numbers (as any mathematician will tell you there is not such thing as a truly random number on the computers). That is OK *rm*  functionality if you ask me.


----------



## ralphbsz (Mar 2, 2018)

It is much more complicated, and quite off-topic.  In a Unix file system, you have to distinguish two things: files, and the names that point at files.  Files can have zero or more names.  Names point at exactly one file (except for short transition periods they point at zero or two, and during those periods nobody is allowed to look at them, locking guarantees that).

In the following, I will use the term "inode" to talk about files.  That's not 100% correct, because there is more to a file than just the inode (which does not usually contain the data in the file), and some file systems don't actually use inodes at all, and sometimes inodes themselves are stored as record in files, but for simplicity I'll use the word inode.  The inode contains all the attributes of a file except the name, for example ownership and permissions, mtime and atime and all that, and most importantly some way to find the data of the file on disk.  It does *not* include the file name.  Inodes are stored on disk.  In the old days, inodes were exactly 512-bytes long (one sector), and were simply stored on disk, using the same space allocation method that files used when they needed a sector.

File names, on the other hand, are stored in directories.  You can think of directories as objects in the file system that are just like files, except their content is specially structured: they always consist of directory entries.  For simplicity, we can assume that a directory entry contains a name (relative to the directory it is in, so /home/fred/hello.c would have "hello.c"), a flag whether this entry is a file or a directory (or some other strange entity like a soft link or device or other things we'll ignore), and if it is a file, a way to find the inode.  Most file systems use an inode number for that (not all do).  You can really think of directories as nothing but special files containing lists of directory entries (early file systems implemented them that way).

Let's go through the life cycle of files to get a few examples.  You open a file that already exists by name: You look up the name in the appropriate directory, find the inode number, look at the inode to do some permission checking, find the data, and read or write it.  Boring.  You create a new file that doesn't exist: you add a new directory entry to the appropriate directory, you create an empty inode and write it to disk, and then start writing (or reading after you have written something).  So far it's easy.  Notice that I have deliberately not talked about deleting a file!

The complexity happens because of two nasty things in unix file systems: hard links and temporary files.  Let's do hard links first.  A file already exists, and has exactly one name (the one you created above).  You want to add a second hard link to it: You find the appropriate directory, add a new directory entry to it (with the correct name), and make it point to the same inode.  So now you have two names (two directory entries) that point to the same inode (the same file)!  That fact has to be recorded somewhere (you'll see in a moment why).  That may sound confusing (and I personally hate it, life would be easier if there was a 1-to-1 correspondence between inodes and names), but it works.  The important thing to remember: an inode (the body of the file) can have multiple names that point to it.

Let's do the first example of deleting a file.  Let's take a file that has many names (many hard links): We can delete one of the names (with the unlink system call), and exactly nothing happens.  Superficially, all unlink() does is to remove a directory entry that points to a file.  This sounds like "delete" is broken on Unix file systems, and they will leak disk space, but we'll fix that later.

The crazy thing is creating a temporary file.  That's done like this: You create a new file (new directory entry, new inode), and record that all.  That new file now has one name.  Then, while the file is still open, you delete the file (with the unlink system call).  But the file is still open, so the inode can't go away, but the directory entry is wiped out.  At this point an inode exists, but there are zero names pointing to it.  You can read and write the file, but nobody else will ever be able to find this file again.  Now, what prevents that inode from being harvested, overwritten, or destroyed?  The fact that the file system remembers that 1 user process has it open.  What do we learn from this?  The file system has to keep track of how many names a file has, and whether it is open.

Now let's go completely crazy, and build the super-secret communication channel that nobody can find.  Adam and Bob are two processes on a Unix system, and they want to communicate, without Eve else listening in (don't worry, the thing I'm about to build is actually totally insecure, it's intellectually no more complex than two cans and a string).  Here's what they do: Adam creates a file, and then Bob opens it.  Now the same file is open from two processes, and they can mutually see what each other have written, by reading.  Once they know that communication is working, one of them unlinks it.  Nobody else can find the file (without a name, you can't get to it), so Eve can't spy on them.  The two of them chat for a while, then both close the file.  (By the way, the names Adam/Bob/Eve are the standard names used in the cryptography literature; and since Eve has a brain (she's female after all), she could just have opened the file at the same time Bob did, and read everything the two idiots were talking about.)

So let's ask the question: Those temporary files, when can they be actually deleted?  So here is the answer to how deleting a file really works: unlink'ing a file only removes a name to a file, as we said above.  All inodes have to keep two counters: a counter of how many directory entries point at this inode (is usually 1, but as we saw above it could be 0 for a temporary file, and 2... for a file with multiple hard links), and a second counter of how many processes have this file open right now (this is zero for most files, and in fact once the file is open there is a data structure in kernel memory that describes the open state).  Whenever a file is closed by a process, the kernel can check: if both counters (both the number of names and the number of open instances) have dropped to zero, it can actually wipe out the inode (in practice, they are typically not wiped out, just unallocated and the space becomes available for reuse).

So finally, I can explain how a normal `rm` works: Superficially, it just unlink's a directory entry; but in the normal case (of a file that has a single name and is not open), that happens to also wipe out the inode, and in the process free all the data blocks on disk that are in use by the file.  It is actually quite a complicated process.  The important thing to remember is that the second part of that operation (wiping out the inode and freeing the data blocks) may not happen at the time of rm and unlink, but later.  By the way, I don't want to discuss right now what happens if the computer crashes in the middle of any of these operations, but if you think about it just a little bit, you'll see why fsck exists.

You talked about shred'ing files and overwriting them three times with random bits.  Some file systems can do that when the inode is wiped and data blocks freed.  But the reality is much more complicated.  For example, one can truncate a file to zero size, and some file systems even have the ability to punch holes (unallocated areas) into files.  Should they be shred'ed or wiped such?  Interesting policy question.

(By the way, if you enjoy complex puzzles: Go through the above logic for when the open count and the name count get incremented and decremented, but now with snapshots, and files that can be read from snapshots, and snapshots that can be deleted.  There is a reason that snapshots were added to file systems only after the field had about 20 years of experience.)

In most cases, immediately after an inode is wiped (meaning usually just unallocated), the inode is still on disk without having been overwritten, and the data blocks are probably also still there.  And the directory entry (in the directory itself) is typically also not completely removed, just flagged as "free, can be used for something else".  So if you crash the machine immediately afterwards, and use an offline data recovery tool, you can probably find the inode and data, and perhaps even most of a directory entry for it.  But at this point there are no consistency guarantees: the file system is heavily multi-threaded, and some other thread could have already started using the same disk blocks for something else.  Matter-of-fact, modern file system do not actually implement all that stuff I explained above, because doing all these steps with real IO would be ludicrously slow.  Instead, most of these operations happen in memory, with complex data structures and locking.  With journaling, transactions, and soft updates, the actual writing of these things to disk happens before, after and instead of the simple outline I gave above.  So now some tools come in (either offline tools or even worse online tools like lefsha described), and think they can look at the disk, and modify the state of file system internal structures, or write to disk, and most of the time chaos will happen (it may actually work some of the time).  If undelete were easy, file systems would implement it.  It is hard, and a lot of work.  And the little tools that claim to do the work are usually unsafe, because they are not correctly integrated.


----------



## ralphbsz (Mar 2, 2018)

Finally: You can safely assume that organizations like the NSA are capable of retrieving the data from your disk, even after you deleted a file, and even after you have overwritten it 3 times with random bits.  If you look at scanning electron microscope pictures of tracks, the information is out there, on the edges of the track (and on SSDs in the voltage levels of the capacitor that stores the bits).  Go to  storage conference, or go drink a few beers (or stronger stuff) with people who work for Seagate/Hitachi/Toshiba/WD/..., and you'll hear stories.  On the other hand, for the average home user, you can assume that other average users won't be able to read your disk after rm'ing a file.  And even the normal data recovery companies (who charge you a few thousand dollars to get your data back, after a head crash, or after you rm'ed a file by mistake) won't have much luck if the data has been overwritten even once.  For normal safety requirements, the following command is sufficient: "dd if=/dev/null of=/dev/daXXX ...".  Another good way to secure your data is self-encrypting drives (which require cryptographic authentication to read the data).  Unfortunately, SEDs are also very hard to use, and few systems support them well enough to be practical.


----------



## gkontos (Mar 2, 2018)

lefsha said:


> Do you really wish to say, that Windows 3.11 is more advanced than today's FreeBSD?????? - OMG...



I think they meant that is a more suitable OS for you.


----------



## Crivens (Mar 2, 2018)

I remember there was a file system without a delete: the bullet server in amoeba. That one was aimed at WORM media for storage. Another research OS with interesting concepts.


----------



## lefsha (Mar 2, 2018)

k.jacker said:


> You can still `/bin/rm/[I]filename[/I]` if you really want to remove a file.
> I actually like that, I'm going to fine tune that a bit



Great Idea. Unfortunately in my case I hadn't ran 
	
	



```
rm something
```
.
It was totally different tool, which did something totally unexpected.
No one can control what exact method is used by the certain tool for files deletion.
It can be easily direct system calls.

I tied to explain once again my post was not about technical issue with a file system!


----------



## lefsha (Mar 2, 2018)

ralphbsz said:


> Finally, let me summarize the discussion.  Undelete tools exist for some file systems.  They are all unreliable and dangerous, some more some less.  Under human manual supervision with an expert in file system internals at the control, you can sometimes do better than automated undelete tools, but not always.  Many modern file systems support some notion of snapshot, ZFS definitely does.  Intelligent use of snapshots removes the need for undelete tools.



Can't quote the whole long message. But unfortunately you are wrong in any aspect of this discussion.

1. First it was a personal issue about how to deal with questions many people are asking. Look across internet, look out in this forum and you will find how many people are asking about the same problem. And even more. It doesn't matter what people are asking about. They do have problem, therefore they are here to look for support! Instead of helping them some persons are trying to blame them for doing something wrong.

Therefore I put my self in exact the same position trying to teach those teachers who blame other people by demonstrating they don't know the facts and they should not teach or blame other people. Instead they might HELP people and they can't just stay aside.

*Is it so difficult to understand?*

2. Nice you have confirmed, that undelete functions or programs do indeed exist for many file systems. They aren't every day tool. Nobody claim the opposite. It's not something you run on daily basis, but those help to save your ass if something bad and unexpected has happened.
Which means they do not only exist but also quite helpful !!!

*That makes the question, whether a certain file system does have such a tool fully adequate!* - That means at least there is no reason to blame a person who is asking such a question!

Am I right?

3. Who cares whether the tool is dangerous if it can save 1-2-3 hours of hard work??? The outcome is simple it does work or not. There are no
different results. Undelete works only with not occupied blocks. And people who wrote all these thousands of programs aren't more stupid,
than those who pretending of being experts! Real experts would never say such a bullshit and would never blame people in trouble!


4. In particular case of ZFS snapshots are helpful, but they don't remove the need of undelete tool. I have demonstrated that with my personal case.

*Please note, none of you so called experts told me how should I behave to avoid undesired file delete which has happened to me 2 days ago!*

That means - none of given arguments are working in the real life. In theory it sounds nice, but real life is more complex than that.
A reminder. In my case it was a third side tool, which should delete the database 
	
	



```
pg_dropcluster
```
 That action was expected and desired. But in addition to that it has deleted the config file for database I spent long time with. As we all know usually the system updates or even package removal do not touch configuration files or user's settings. In most cases it won't delete those files at all, in other cases it you have to provide a special option for that to happen. Well, I have not expected from PG to behave in that bad manner.

At the end it doesn't matter, because any program could delete files from the system as long as it does have permissions to do it.
There is no insurance against it. And that is only an example, that a user can delete files without knowing it or totally not intentionally.
And that means there should be a solution against such a bad thing. If file would be temporally deleted and kept on until next snapshot
will be issued, that would be a complete and proper solution! That is how I would address it from the technical perspective.

Instead we do have stupid discussions about blaming users for their mistakes... - Quite bad attitude I would say.

5. About Linux... It seems you are another expert who made no difference  between a kernel and OS. It does seem also the word Linux
works like a red tape on many people like you. I haven't blamed FBSD or ZFS or Linux in that thread!!! I did mention them, but wasn't blaming them.
I did mention Windows3.1 Does it mean I do admire it or blame it? Nothing of above. That is no point. The major and only point was people.
I do blame them! And that it indeed the truth. People should be more friendly to each other and don't attack people who are just asking questions!!!

Was it to complicated to understand?


----------



## lefsha (Mar 2, 2018)

Eric A. Borisch said:


> It appears (I haven't used it) that hammer has some ability to undelete: https://www.dragonflybsd.org/cgi/web-man?command=undo .. it looks to rely a nightly (cron-initiated) prune of deleted files, but there is some ability to recover until then. Certainly an interesting design, and not one I've seen before.



Very appreciated! That is absolutely ideal behaviour and every one should learn from it.

Mainly I was not asking for help in my particular situation. I was just looking for a solution and would not say a word - just read, until I found
other people being blamed for asking questions! And that was my reason to step in.

The suggestions to do snapshots after the file was deleted is like saying at the funeral the dead man should not be smoking.
It's totally impolite, impractical, doesn't help to anyone and sounds ridiculous. Although the saying people should not be smoking is quite reasonable, helpful advice.


----------



## lefsha (Mar 2, 2018)

I was put on premoderation. Congrats - you have find the evil. It's a wonder how blind people are.


----------



## Sensucht94 (Mar 2, 2018)

I think that FreeBSD can't be blamed for a ZFS feature. I would be like blaming Linux for Btrfs having allocated huge metadata in raid1 abd wasting space. FreeBSD has its own UFS2 version, which is rock solid (and allows data recovery).

Whether or not this "permantly-rm" ZFS feature is a pro or a con, I guess it highly depends on different situations, users, targets, implementations. I'm sure there's a large slice of FreeBSD users wanting this file system stay the way it is and keep following tge same direction it has so far.

Anyway, for anyone wanting to attempt recover a file on UFS, XFS, EXT* and other filesystem, I strongly recommemd checking out foremost(), sysutils/foremost


----------



## Crivens (Mar 2, 2018)

lefsha, when you  shoot yourself in the foot, do you blame the gun?

Some very experienced people told you what went wrong, that there is no magic wand to make it all right and what to do to not repeat the mistake. Nobody here will help you save your face, but they help you learn from mistakes.


----------



## Eric A. Borisch (Mar 2, 2018)

lefsha said:


> But in addition to that it has deleted the config file for database I spent long time with.



From the man page for pg_dropcluster: (emphasis mine)


> This program removes all files that belong to a given PostgreSQL cluster; that includes the data directory, the log file, *and all configuration files* that were created by pg_createcluster(1).



Granted, many (most? [citation needed]) tools won't  delete your config files, but this one explicitly says it will.

That said, if all you are trying to recover is your config (text) file, AND you remember a string from the file (hopefully fairly unique) AND you didn't have ZFS compression enabled, you can just run grep on the raw device and see what turns up. Run it with '-a' and likely '-C <n>' where n is >= the number of lines in your config file. Note this may take hours depending on the device.

EDIT: AND it is a single device pool. (Or a small file on a multi-dev pool.)


----------



## ralphbsz (Mar 2, 2018)

lefsha said:


> They do have problem, therefore they are here to look for support! Instead of helping them some persons are trying to blame them for doing something wrong.


You did something wrong, you ran a command that destroyed some files which you didn't want to destroy.  That's a sad fact, and you have to accept the blame for it.

Now, this type of mistake happens all the time, to everyone.  I've deleted files that were still needed zillions of times in my life.  Mistakes happen.  And that's why people are careful.  For example by taking a snapshot of the file system right before doing something that might be dangerous.  Or by taking a backup right before.  That can be done with big, complex automated backup system (I have one on my home server which takes a backup every hour).  Or it can be done manually, for example "cd /usr/local/etc; tar -cf /tmp/before.tar .".



> Nice you have confirmed, that undelete functions or programs do indeed exist for many file systems.


I would not agree with this statement, because "functions and programs" implies something that meets the quality standards one expects of system programs.  For many file systems, there exist after-the-fact hacks which are not well integrated with the file system, and that are unreliable and dangerous.



> That makes the question, whether a certain file system does have such a tool fully adequate!


If undelete was easy to write (it is not, it is very hard), and if it was a good investment of time (it is not, since better practices exist, like snapshots or backups), then it would have been written.  People in the storage field have known this for many decades.  As an example: The VAX was introduced in 1977, and it ran the VMS operating system.  The file system on that machine kept older versions of files around, just to protect against files being inadvertently modified.  That was 40 years ago!  Remember the syntax: file names had the format "[fred.source]hello.c;7", meaning that user fred's home directory had a subdirectory called source, in which his hello.c program had been written 7 times.  But: Even VMS did not have an undelete command!  If you simply said "del hello.c;", it would delete the most recent version, but that would mean that the next-older version would become the current one (in effect reverting to an older one). To actually delete a file completely, you had to say "del hello.c;*", which deleted all versions, and now it was gone.



> Who cares whether the tool is dangerous if it can save 1-2-3 hours of hard work??? The outcome is simple it does work or not. There are no different results.


No. Badly written undelete programs leave the file system in an inconsistent state.  On-disk data structures of file systems consist of many interrelated parts, and have many invariants that need to be maintained.  For example, if a sector on disk is in use in a file, that fact needs to be reflected in an allocation bitmap, which is usually kept both on disk and in memory (not all file systems use explicit allocation bitmaps, real-world ones are way more complicated than that, with extents and pre-allocated lists, and usually only a part of the allocation bitmap is kept in memory).  If one uses a sector for a file that is being undeleted, that needs to be reflected in the allocation bitmaps.  And those updates need to be done under lock (because computers are by their nature multi-threaded and things happen in parallel), and using the correct ordering (so if a crash occurs in the middle of the operation the next restart or fsck puts the on-disk structures back into a consistent state).  It would be very difficult for undelete program that is implemented outside of the file system (as a separate utility) to use the correct locking (since it can't reach into the file system to acquire and release locks), nor to do the ordering right.  I keep repeating the same thing: to do undelete right, it would need to be done inside the file system implementation.  It isn't inside, therefore it isn't being done right.



> Real experts would never say such a bullshit and would never blame people in trouble!


The real problem with your posts is not that you are asking for assistance after an unfortunate mistake.  The real problem is these types of broad attacks on everyone.  If we all are not real experts, then why do you even bother to ask us for assistance?  I'm sorry, but your posts are logically inconsistent.  You also keep saying things like "People should be more friendly to each other and don't attack people who are just asking questions!!!" (with triple exclamation marks).  Sorry, you are not just asking questions.  You are shouting at people, with triple exclamation marks, telling them what they should do, and you are telling them what is wrong with them in your mindset.



> In particular case of ZFS snapshots are helpful, but they don't remove the need of undelete tool. I have demonstrated that with my personal case.


If your configuration data for "pg_dropcluster" had been in ZFS, and if you had done a "zfs snapshot" before, all you would have to do is to go back to that snapshot (for example by copying files from the snapshot back to the live file system).  Really, no need for an undelete tool.



> ... In most cases it won't delete those files at all, in other cases it you have to provide a special option for that to happen. Well, I have not expected from PG to behave in that bad manner.


Read the documentation before using the command.  By the way, I have *NO* idea what pg_dropcluster does, but I know that one has to read its documentation before using it.
And see above: Take a snapshot of all your configuration data.  Or back it up: look at the tar command I gave above: It copies all of the /usr/local/etc hierarchy, just in case something gets damaged.

To summarize this post: A reasonable discussion would have been that you asked whether there is an undelete tool for ZFS.  The answer is that there is not, for unfortunately good technical reasons.  Then you could have asked what other options exist to safeguard against inadvertent damage to configuration files, from running a command whose side-effects were not understood by the person running it.  The answer is to take snapshots before, or back up the configuration files.

EDIT: Because of these posts in the last few days, I've been thinking about how one could add real undelete functionality into a file system for fun.  I think it can be done, but it requires considerably trickery.  One would have to use a versioning mechanism (see the VMS example above), and combine it with a non-Posix interface to get to the deleted files.  Elephant's trick (pound sign in file name) unfortunately violates Posix, but at least it's a starting point.  Organizing deleted files in the metadata is possible, but rather nasty; one would have to create a system of shadow or zombie flags in various places to mark entries such as inodes and indirect blocks; and then every bit of code in the file system would need to check these zombie flags.  The real issue is that eventually, undeletable files really do need to be deleted.  And the order of doing that is highly non-trivial.  Alistair's paper "When to forget in the Elephant file system that never forgets" only exposes the tip of the iceberg; to do the same in a modern production system with high performance would need good insight into use patterns, to create the heuristics of when to finally delete.  Doing this would probably take several person-years of work, but it's the kind of thing that could get "best paper award" at a conference.  Might be a good thing for a group of graduate students to take on as a PhD thesis group project.


----------



## Crivens (Mar 2, 2018)

VMS used a `purge` command to prune old versions. Great. Now I have been mugged strolling down memory lane...


----------



## ralphbsz (Mar 3, 2018)

Crivens said:


> I remember there was a file system without a delete: the bullet server in amoeba. That one was aimed at WORM media for storage. Another research OS with interesting concepts.


Thank you for reminding me of that!  I had completely forgotten about Amoeba, in spite of the fact that I had to present it in our departments "journal club" once.  By the way, the principal father of Amoeba (Sape Mullender) has also written a really nice textbook about distributed systems, which I have upstairs on the bookshelf.  Good reading material.

And the "purge" command on VMS is exactly a symptom of the problem that one gets in a file system that has too many snapshots: space management becomes a terrible chore.  VMS dumped the problem on users, which means people had to run "purge" manually whenever the sys admin told them that we're about to run out of disk space.  Later VMS versions added the ability to automatically delete older versions, and limit the number of versions per file to a fixed number.  Today, the problem of taking automatic snapshots, and decimating or deleting them as a space management technique, tends to be integrated into HSM; that's a big and complex field in and of itself (not my specialty).

And by coincidence, by going through some older paperwork today I discovered that I actually hold two patents on how to implement snapshot in file systems.


----------



## Snurg (Mar 3, 2018)

I now feel I understand better why all Windows versions since 4 have that trash can like Macs. (Imho the advice to use 3.11 is wrong insofar)
(At least they offer shift-delete to bypass the annoying trashcan )

But I really think FreeBSD should make more clear that desktop usage is not officially supported, but instead volunteer work.
It's like concentrating on core businesses - drop things that gain little and focus on a few things to make good.

So people like me who are whining because things like suspend/resume fail (even though they were implemented because the foundation sponsored this) know it has no use to whine.

The suggestion to drop stuff more aggressively has its good sides. There were recently a few people that honestly expected bluetooth to work on FreeBSD.
Dropping this for example would be not bad.

Dropping KDE, Gnome, Xfce, Cinnamon etc too would be helpful probably, too.
Better 3000 well-running packages than boasting 30000 packages of which 27000 are crap...


----------



## Snurg (Mar 3, 2018)

Another thing I just saw in another thread... there are people using sudo.
Is there a particular reason why to use sudo, in the light that one can do `sudo rm -rf /` ?
(Even with sudo one needs to turn on ones' brain...)


----------



## ralphbsz (Mar 3, 2018)

I don't understand your question.  One needs to be root to do many system administration tasks.

If one is logged in as a normal, non-privileged user, one has to use sudo to do that.  That seems to be safer than the alternatives, which is is to outright su and become root, or even log in directly as root.


----------



## Mage (Mar 15, 2018)

lefsha said:


> 1.
> And I insist it's a bad attitude trying to teach people, who were not asking any one to teach them!!!
> Is it so difficult to understand? Reading this forum or Stack Exchange/Overflow I am always wondering where those people come from?



I feel for you, but this is nothing to do with either ZFS or FreeBSD or computers but the apes you called people.

For example, if you open a topic in which you ask whether you have the right to refuse to visit your apartment to your landlord on a certain date not suitable for you, and what the legal/financial consequences can be, and you state in your OP that letting they come in on the specific date is not an option, in the first 20 responses, about 15 people will either ask you what your mental problem is, or offer you ideas how to let the landlord come in (by giving keys to someone else), or explaining it to you how impossible it is that you could have any reason to refuse the visit.

Maybe one person of twenty will tell you a helpful hint.

Considering the average habitants of this planet, I find the FreeBSD forum helpful in cases. (The case when I asked how to encrypt a production server is not included.)

I believe the psychological reasons might be jealousy or self-justification or something similar. If they prove you were wrong or you have less values than you think, they feel better about their lives.

You are also right about how the 99.9% of deletions happen.

I don’t like snapshots. I do have backups. I usually have backups on multiple machines, but I just deleted a backup of something temporary but needed. I figured it out one minute later. That’s why I found this thread.

I have another backup but that’s not the last version. As you see, although I had multiple backups, I still lost data.

Yeah, it sucks.


----------



## Mage (Mar 15, 2018)

ralphbsz said:


> I don't understand your question.  One needs to be root to do many system administration tasks.
> 
> If one is logged in as a normal, non-privileged user, one has to use sudo to do that.  That seems to be safer than the alternatives, which is is to outright su and become root, or even log in directly as root.



False security is worse than no security. When sudo is used in a script, you may end up running things as the root you don’t want to run as the root.

It was probably implemented due to people who would have a PTSD after logging in even once as a non-root user or running a web application with any other user than the root.

For me, it works well to have a red (ASCII) prompt for the root, and some blueish prompt for the normal user, no sudo, not running around as root unless it’s needed.


----------



## vince66 (Apr 30, 2018)

Very very very important thread !!!

Please, let me the possibility to reply .... I need to correctly plan it first.

Bye bye !!!


----------



## cld_tech (Apr 27, 2019)

Didn't try it cause it's Windows only and there's no words to express my feelings for windows. 
But if there is some great details in the page too.
https://www.klennet.com/notes/2019-01-05-zfs-undelete.aspx


----------



## FreeSoftwareServers (Apr 8, 2020)

I too am unimpressed with the "Holier than thou" attitude of many people in this thread and on the internet in general, data recovery programs have existed for almost every filesystem, and many people don't explore that avenue until after disaster strikes, they google/ask in a forum for support. The saying "If you don't have anything good (productive) to say, don't say anything at all" comes to mind.... Regarding all this ZFS recovery I'd like to say I have had success recovery files using UFS Explorer to recover files from ZFS partitions. I highly recommend the software, you can tell it's well written and it worked great for me!


----------



## djjaeger82 (Dec 26, 2020)

Which version of UFS Explorer did you use?  And what type of ZFS volume were you recovering from?  Sadly I find myself in need of some data recovery on ZFS volumes after accidentally deleting a VM that removed to VMDKs that were associated with other VMs...One of my ZFS volumes is a simple mirror (SSD 512gb x2) and the other is a RAIDZ1 (4x 8tb).  UFS Explorer Raid Recovery v8.10 was able to read and see both deleted VMDKs, and upon inspection said there were 0 errors on either file, and both were recoverable.  Unfortunately after purchasing a license, the VMDK on the simple mirror volume started to recover but then errored out 50% of the way thru for "insufficient system resources to complete the operation".  The VMDK file on the RAIDZ1 volume won't even start recovering, and says "unable to read source file".  I'm hoping this is something to do with the new v8.10 version they just released with a rewrite of their ZFS code.  I hate to start all over again with another scan from scratch since it took nearly 3 days on the RAIDZ1 volume.... 

Any tips/tricks or other utility suggestions would be greatly appreciated.  (please don't be another one of those, you should have had snapshots and proper backups people, I get it, I screwed up, I will set this up going forward, but I'm looking for help here and now with my current situation)...

I've also been reading a little about ZFS transaction rollback, is that a possible avenue too?


----------



## ralphbsz (Dec 27, 2020)

Transaction rollbacks are internal to the ZFS implementation. AFAIK, transactions are super short lived (across a handful of IOs), and are committed within a few dozen milliseconds, so there won't be any rollback opportunity once committed. For details, Kirk McKusick's book (Design and Implementation of FreeBSD) has lots of explanations; or go take his class on ZFS internals.

I have zero experience with "UFS explorer". No idea who these people are. Judging by their address, I suspect that they are the former ReiserFS implementation group, which became unemployed after Hans Reiser's unfortunate little legal accident (he murdered his wife, and got caught, oops, hate it when that happens). If you bought a license, you should contact them for technical support, since the tool is clearly not working correctly.

All recovery tools are voodoo. They usually barely work or don't work. With expert hand-holding (you get someone who knows ZFS internals to go through each step), the probability of succeeding will be much higher. Consider the cost of that: An expert who can do that will costs hundreds per hour (dollars, euros), and it will probably be many hours of work to do this. I've had to do this for file systems that I was a developer on a few times, and typically a large data recovery operation is several days of work (I remember a particular incident that took 3 days, working 18 hours per day, including a weekend ... but the customer was very big and important). Do you think the data is worth that much to you?

Instead of being obnoxious and giving you advice, I will tell you a joke. There are two kinds of computer users: those who religiously do backups (and use snapshots), and those who haven't lost data yet. And: Happy Holidays!


----------



## Mirror176 (Dec 31, 2020)

Guess I'm a third user from that joke; I don't religiously do backups and have lost data (backups or not, still lost it). Backups can die and can become corrupted. Backups don't matter if you haven't tested that you know how to get into that data; document that process because you don't want to fumble on old memory or lack of knowledge when it is time to put things back because you are probably distracted by bad things having already happened. I did a rm as root once on this very computer; the damage done was basically equivalent to ~/.*  in part of the command from my normal user's directory so damage was not controllable form not being root. I got back what little I wanted successfully from grepping an offline post-delete copy of the drive and moved on with my life; don't remember if I had a relevant backup back then or care but it was fun to 'try' to get my data rather than important.
  In this thread there were a few utilities mentioned: grep works if uncompressed+unencrypted+unfragmented+content is known. photorec can help automate an attempt if it can detect the desired content type in the same circumstances. There is at least one utility claiming undelete for zfs though it ran under Windows and when I head of it Windows doesn't use zfs itself; may be worth it to some people if it gets the data though. Didn't notice zdb being mentioned but it is supposed to be the tool to be able to walk a zfs dataset. Transaction rollback 'may' get somewhere with it depending on the changes the pool has undergone; thought it was 256 versions before the structure blocks reuse (can't recall proper names currently) at a given level (but doesn't guarantee your file's contents blocks weren't already reused). I tried to browse some of that data a little bit once but I think it was a large write on a drive without much space left so likely wouldn't mean much and yes multiple structure versions were getting used up on the one large file's write if I recall.


----------



## cederom (Feb 10, 2021)

https://www.ufsexplorer.com/ufs-explorer-raid-recovery.php ?

https://www.freshports.org/sysutils/testdisk/ <- `photorec` ?


----------

