# ZFS Question - How do I remove excess file copies



## rajl (Jan 15, 2012)

I recently built a FreeBSD 9.0 box to use as a fileserver with ZFS.  I chose ZFS for its data integrity and self-healing features.  ZFS really is the "killer app" for FreeBSD if there ever was one.

After much initial research, I built the system and created a zpool mirror spanning two 1TB hard drives named "zdrive."  Within zdrive, I created a zfs filesystem named "home" and set copies=2.  I erroneously assumed that I needed to have ZFS create and store extra copies of the files that I saved in order to take advantage of the self-healing properties of ZFS.

I now know that due to the mirror, zfs will self-heal by replacing the corrupted copy on one disk with the non-corrupted copy on the mirrored disk.  There is no requirement to have copies=2 on a mirror for zfs to self-heal.  If both disks have corrupted copies, something so catastrophic has happened that I will be restoring from an off-site backup anyway.  

What is now happening is that I am needlessly saving 4 copies of each file (two copies to each drive in the mirror).  This is needlessly redundant for my purposes, and it effectively halves the capacity of my 1TB drives to 500GB.  I am running out of space *fast.*

My question is, how do I fix this without nuking the whole thing and starting from scratch?  I tried to remove the excess copies to no avail by issuing the command "zfs set copies=1 zdrive/home".  While that eliminates saving redundant copies moving forward, the redundant copies already saved are still on my zfs filesystem.  I would like to reclaim this space.

Does anyone have any suggestions?


----------



## gkontos (Jan 15, 2012)

You can create a snapshot of /home and send it to a backup device. You will then destroy your initial pool and recreate it. After that you are ready to receive back the snapshot.

A few things you have to watch out for. When ZFS properties are changed, they only affect new data. Therefore, you might change the copies but as you saw by yourself this didn't affect your existing data.

When you send the snapshot don't use the -R switch because you don't want the properties to be also sent. See  zfs() for more information.

If you don't have the necessary space for this procedure, you can still do it by degrading your mirror and using the second disk as a new pool for the transfer. However, you should already know that a mirror is not a backup strategy.


----------



## rajl (Jan 15, 2012)

Thanks for the direction.  I read the man page and think I know how to do what you recommend.  Could you please double check what I plan to do to make sure I understand it correctly.

After reading the man pages, I believe this will be a 6 step process:

Step 1: Create zpool on backup device named "zbackup."
Step 2: Issue command - zfs snapshot zdrive/home@backupsnapshot
Step 3: Issue command - zfs send zdrive/home@backupsnapshot | zfs receive zbackup
Step 4: Issue command - zpool destroy zdrive
Step 5: Issue command - zpool create zdrive
Step 6: Issue command - zfs send zbackup/home | zfs receive zdrive

I have two questions about these steps.  First, should there be a seventh step in middle where I create a snapshot on the backup device before resending it back to the recreated pool?  Also, even if the copies property is not replicated with zfs send, will the additional copies of the files still be sent?  I'm a little confused about the difference between data and properties.  When the copies property is set to copies=2, are the additional versions of the file that are saved considered a "property" or are they "data."

Thanks for your help, time, and patience.

I am sorry for the confusion.


----------



## gkontos (Jan 16, 2012)

rajl said:
			
		

> I have two questions about these steps.  First, should there be a seventh step in middle where I create a snapshot on the backup device before resending it back to the recreated pool?



No, because you sent a snapshot of your pool. You will send back the same snapshot.



			
				rajl said:
			
		

> Also, even if the copies property is not replicated with zfs send, will the additional copies of the files still be sent?  I'm a little confused about the difference between data and properties.  When the copies property is set to copies=2, are the additional versions of the file that are saved considered a "property" or are they "data."



They are properties of the pool. Even if you use the -R switch when sending, only one copy will be sent over. The difference is that you would eventually end up by storing two copies on the receiving pool. That is why I said that you shouldn't use it.


----------



## SocialHaze (Jan 16, 2012)

You can also copy your files into the existing pool and erase the old copies, assuming you have enough space left.


```
zfs set copies=1 zpool
cd zpool
mkdir copy
cp -pR * copy
find -d -name !copy (or whatever)
```


----------



## rajl (Jan 16, 2012)

I just had a (maybe) brilliant idea?  Would this work:


```
# zfs create -o copies=1 zdrive/temp
# mv /zdrive/home/* /zdrive/temp/
# zfs set copies=1 zdrive/home
# mv /zdrive/temp/ /zdrive/home
```

Basically, I'm creating a new file-system with copies=1, moving everything over to it, setting copies=1 on the old file-system, then moving everything back in order to reclaim space.  Since zfs is copy-on-write, I could see this being extremely fast as no data is actually changing location.  Also, I think this would remove the properties attached to the previously saved versions on the file-system.

Am I crazy, or would this work to solve my problem?


----------



## SocialHaze (Jan 16, 2012)

Datasets are filesystems.  The operation would be lengthy but you've grasped the concept that you need to rewrite the data in order to eliminate the duplicated data.

According to your paths and granted the space, you could:


```
# zfs set copies=1 zdrive/home
# cd /zdrive/home
# mkdir temp
# mv * temp
# cd temp
# mv * ..
```

But essentially the process is the same: the files need to be copied, the duplicated files removed along with their shadow copies and the copied files restored to their intended path.

I'm actually not sure about ZFS' internals, though I would hypothesize that each ZFS' version of an inode points to two physical registers on the drive.  There's likely a way to rewrite the ZFS inode table, though I also think that this could cause an interlacement of data on the drive.  As much as possible, I like my data to be contiguous, or at least not shredded in such a way.  This is likely the reason why it's preferable for the issue to be resolved in this manner.


----------



## rajl (Jan 16, 2012)

Sceak,

I did exactly as you suggested, based on my idea.  The operation took about 5-seconds and did absolutely nothing to remove the shadow copies.  My guess is that it merely shifted the inode pointers of the directories inside zdrive/home to /zdrive/temp.  Since the operation did not touch the files inside the directories, it did not rewrite data and thus no shadow copies were removed.

I will try the other two suggestions and report back which one, if either, worked.


----------



## phoenix (Jan 16, 2012)

I'm almost positive that you will have to:

create a new pool (or even just format an external drive as UFS)
copy the data from the existing pool to the new one
verify the copied data
delete the data from the existing pool (or even just delete the entire filesystem)
set *copies=1* on the filesystem
copy the data back
verify the copied data
delete the new pool
You have to move the data out of the pool, set the zfs property, then move the data back to the pool to force it to write it out with the new properties.

Copy/move inside the pool won't be enough, although crossing ZFS filesystems may do it.  But removing it from the pool entirely will guarantee it.


----------



## SocialHaze (Jan 17, 2012)

rajl said:
			
		

> Sceak,
> 
> I did exactly as you suggested, based on my idea.  The operation took about 5-seconds and did absolutely nothing to remove the shadow copies.  My guess is that it merely shifted the inode pointers of the directories inside zdrive/home to /zdrive/temp.  Since the operation did not touch the files inside the directories, it did not rewrite data and thus no shadow copies were removed.
> 
> I will try the other two suggestions and report back which one, if either, worked.



I was relying on mv's copy/delete mechanism... not sure what happened.  I did the test myself and while a move operation didn't perform as I expected a copy/delete/move fixed the issue on my test dataset.

Edit: A quick look at the source reveals that mv, as the epitome of efficiency, will attempt to rewrite file descriptors rather than perform an actual copy operation when possible... who would've thought?


----------



## throAU (Jan 17, 2012)

^^ well, mv = equivalent of dos RENAME.

I suspect you could perhaps write a script to do something like (psuedocode):


```
for all files
  cp $file $file.bak
  rm $file
  mv $file.bak $file
end for
```

That way you only need the space for 2 copies of your largest file.

Any reason that wouldn't work?  Maybe take a snapshot of your ZFS first?


----------



## rajl (Jan 17, 2012)

What I did was not the best solution, but it was one that worked.  I wish throAU had posted earlier.  His idea is sheer genius and probably would have been the best in hindsight.

I had copied most of the data previously to a UFS formatted external harddrive.  What I did was issue the command [cmd=]cp -an /zdrive/home/ /mnt[/cmd] and let it copy new files to the external harddrive.  I then deleted the filesystem with [cmd=]zfs destroy zdrive/home[/cmd] and recreated it with [cmd=]zfs create  -o copies=1 compression=on atime=off zdrive/home[/cmd]  After that, I issued the command [cmd=]cp -a /mnt/home /zdrive/home[/cmd] to copy everything back.

A few points to clarify:
1) This does *not*guarantee data integrity.  Statistically speaking, the odds of having your data be corrupted during this procedure are slim to none.
2) I used the -n option to avoid overwriting previously backed up files on my external harddrive to save time.  However, I found out that the behavior is to avoid overwriting files with the same name and does not take into account datestamps for when the file last modified.  It is possible, but unlikely, that I may have lost the most recent version of a previously saved file.  Use the -n flag at your peril.
3) I tried to create a snapshot and pipe zfs send into zfs receive.  I then realized I did not have enough drive space to do what I was attempting to do, so I canceled the operation.  Given more harddrive space, it may work.

In hindsight, if I were not so rusty on the command line or with *nix (i.e. if the thought had occured to me), I probably would have done something like what throAU proposed.  Even simpler, since my fileserver only has two users, might be:


```
cp -a /zdrive/home/user1/ /zdrive/home/user1tempdir
cp -a /zdrive/home/user2/ /zdrive/home/user2tempdir
rm -r /zdrive/home/user1
rm -r /zdrive/home/user2
mv /zdrive/home/user1tempdir /zdrive/home/user1
mv /zdrive/home/user2tempdir /zdrive/home/user2
```

Since my problem is solved, I am marking this thread as solved (or if I do not have permission, I ask that a moderator mark it as solved).  Hopefully my experiences help everyone else new to FreeBSD and ZFS.

Thanks for all the help.


----------

