# Incremental ZFS backup errors



## xy16644 (Jan 26, 2014)

I am trying to setup automated backups for my root ZFS pool. Currently I am just trying to get it working by manually running the commands. On my root ZFS pool I have the following datasets:


```
zroot                           /
zroot/tmp                       /tmp
zroot/usr                       /usr
zroot/usr/home                  /usr/home
zroot/usr/ports                 /usr/ports
zroot/usr/ports/distfiles       /usr/ports/distfiles
zroot/usr/ports/packages        /usr/ports/packages
zroot/var                       /var
zroot/var/crash                 /var/crash
zroot/var/db                    /var/db
zroot/var/db/pkg                /var/db/pkg
zroot/var/empty                 /var/empty
zroot/var/log                   /var/log
zroot/var/mail                  /var/mail
zroot/var/run                   /var/run
zroot/var/tmp                   /var/tmp
```

In preparation for the backups I did the following:

```
zfs create zstore/dailysnapshots
zfs set mountpoint=/dailysnapshots zstore/dailysnapshots
```

I changed the mountpoint so that I can access all my snapshots in /zbackups/dailysnapshots.

I then took the first full backup as follows: 
`zfs snapshot -r zroot@YESTERDAY`
`zfs send -R zroot@YESTERDAY | zfs receive -Fduv zstore/dailysnapshots`

I then run the first incremental backup:
`zfs snapshot -r zroot@TODAY`
`zfs send -R -i zroot@YESTERDAY zroot@TODAY | zfs receive -duv zstore/dailysnapshots`

At this point all is great and I don't receive any errors. Except one. I can't see var, usr and tmp mounted in zstore/dailysnapshots. So I thought ok, I'll mount those with `zfs mount -a`. Now I can see var, usr and tmp in /zbackups/var, /zbackups/usr and /zbackups/tmp respectively. This isn't mounted in the correct location so I run:
`zfs set mountpoint=/dailysnapshots/tmp zstore/dailysnapshots/tmp
zfs set mountpoint=/dailysnapshots/var zstore/dailysnapshots/var
zfs set mountpoint=/dailysnapshots/usr zstore/dailysnapshots/usr`

Now I can see everything as I want it in zstore/dailysnapshots and I can see my snapshots in the .zfs/snapshots directory for the relevant dataset.

But when I try to run the next incremental backup with:

`zfs send -R -i zroot@TODAY zroot@TOMORROW | zfs receive -duv zstore/dailysnapshots` I get:


```
receiving incremental stream of zroot@TOM into zstore/dailysnapshots@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr@TOM into zstore/dailysnapshots/usr@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports@TOM into zstore/dailysnapshots/usr/ports@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports/packages@TOM into zstore/dailysnapshots/usr/ports/packages@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports/distfiles@TOM into zstore/dailysnapshots/usr/ports/distfiles@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/home@TOM into zstore/dailysnapshots/usr/home@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/tmp@TOM into zstore/dailysnapshots/tmp@TOM
cannot receive incremental stream: destination zstore/dailysnapshots/tmp has been modified
since most recent snapshot
warning: cannot send 'zroot/var/log@TOM': Broken pipe
```

Why is it saying zstore/dailysnapshots/tmp has been modified when is hasn't? FYI: I have atime turned OFF on all pools/datasets.


----------



## xy16644 (Jan 28, 2014)

One of the ZFS gurus on the forum must have some ideas   

I don't understand why the incremental backups break when I mount all chile datasets?


----------



## gesperon (Jan 31, 2014)

I ran into this issue a while back, if I recall correctly I added `-F` to `zfs receive`. Hope it helps


----------



## xy16644 (Jan 31, 2014)

gesperon said:
			
		

> I ran into this issue a while back, if I recall correctly I added `-F` to `zfs receive`. Hope it helps



Many thanks! I am still stumped by this problem. I shall give you recommendation a try. With this option were all the datasets/filesystems mounted after the incremental backup ran? Were you backing up root on ZFS?

reading zfs() I see it says `-F` does the following:


```
-F
	       Force a rollback of the file system to the most recent snapshot
	       before performing the receive operation. If receiving an incre-
	       mental replication stream (for example, one generated  by  "zfs
	       send -R -[iI]"), destroy snapshots and file systems that do not
	       exist on the sending side.
```

Won't this destroy all my older snapshots? I want to be able to keep all my snapshots so that I can refer to them in case I want to do a restore. I plan on having a year or twos worth of backups.


----------



## gesperon (Jan 31, 2014)

I was backing up a bunch of jails under rpool/JAILS/[jail0..N]. On the destination they were mounted.

I did a quick and dirty test. Seems to work even after modifying the destination (`mkdir /mnt/2/a`):

```
root@fbsd9:~ # zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
rpool                 13.3G  18.0G   144K  none
rpool/1               87.5K  18.0G  87.5K  /mnt/1
rpool/2               87.5K  18.0G  87.5K  /mnt/2
...
root@fbsd9:~ # zfs snapshot rpool/1@a
root@fbsd9:~ # zfs send -R rpool/1@a | zfs receive -Fuv rpool/2
receiving full stream of rpool/1@a into rpool/2@a
received 41.7KB stream in 1 seconds (41.7KB/sec)

root@fbsd9:~ # mkdir /mnt/2/a

root@fbsd9:~ # zfs snapshot rpool/1@b
root@fbsd9:~ # zfs send -R -i a rpool/1@b | zfs receive -Fuv rpool/2
receiving incremental stream of rpool/1@b into rpool/2@b
received 312B stream in 1 seconds (312B/sec)
root@fbsd9:~ # ls -l /mnt/2
total 0
```


----------



## gesperon (Jan 31, 2014)

xy16644 said:
			
		

> gesperon said:
> 
> 
> 
> ...



The man is a little confusing. -F will destroy whatever is on the destination that doesn't exist on the source. If the source have snapshot 1, 2 and 3, the same snapshots are going to be kept on the destination.


----------



## xy16644 (Feb 1, 2014)

I ran the incremental backups last night using my script and your suggestion seems to have done the trick! I'll monitor it over the next few days to make sure but this is great news.


----------



## xy16644 (Feb 2, 2014)

I found an issue with using -F with `zfs receive`. It does delete my older snapshots on the backup pool.

So, how do keep ALL the snapshots that are received to the backup pool AND run incremental backups so that I can't receive the error that the destination dataset has been modified?


----------



## xy16644 (Feb 2, 2014)

I think I may have the incremental ZFS backups working now. Thought I'd share my config to see if I've done it properly.

First I create the datasets:

```
zfs create zstore/dailysnapshots
zfs create zstore/dailysnapshots/tmp
zfs create zstore/dailysnapshots/var
zfs create zstore/dailysnapshots/usr
zfs create zstore/backuptofile
```

Then I change the mount paths:

```
zfs set mountpoint=/dailysnapshots zstore/dailysnapshots
zfs set mountpoint=/dailysnapshots/tmp zstore/dailysnapshots/tmp
zfs set mountpoint=/dailysnapshots/var zstore/dailysnapshots/var
zfs set mountpoint=/dailysnapshots/usr zstore/dailysnapshots/usr
zfs set mountpoint=/backuptofile zstore/backuptofile
```

Then I take the first full backup and mount all filesystems:

```
zfs snapshot -r zroot@a
zfs send -R zroot@a | zfs receive -Fduv zstore/dailysnapshots
zfs mount -a
```

Take the first incremental backup:

```
zfs snapshot -r zroot@b
zfs send -R -I zroot@a zroot@b | zfs receive -Fduv zstore/dailysnapshots
```

Note that `zfs recv` used -F.

Then I do the next incremental:

```
zfs snapshot -r zroot@c
zfs send -R -I zroot@b zroot@c | zfs receive -du zstore/dailysnapshots
```

Here is the bit that is different: For the second (and all subsequent) incremental backups I REMOVE the -F option from the `zfs recv` and the backups seem to run fine with no errors.

I'm going to keep my eye on them this week and see how it goes. If you use -F with the first incremental backup it deletes all the snapshots on the backup pool which is not what I wanted.

Comments are welcome.


----------



## xy16644 (Feb 3, 2014)

Argh, after running my backup script last night I got the dreaded error again:

```
cannot receive incremental stream: destination zstore/dailysnapshots/usr has been modified
since most recent snapshot
warning: cannot send 'zroot/usr@2014.02.03': Broken pipe
```

What am I doing wrong with the incremental backups? How do other people backup root on ZFS using snapshots to another pool on the same machine?

Is it even possible to only have one snapshot on the source pool but have many snapshots on the destination pool? I only ever want to have one or two snapshots on the source pool but I want to have up to a years worth of snapshots on the destination pool. Is this possible?

I seem to be running out of options here.


----------



## xy16644 (Feb 4, 2014)

Can anyone help me here?

I just can't seem to ge this right. All I am trying to do is use incremental backups with my root ZFS. 

Is it possible for me to only have one snapshots on the live root ZFS pool but have up to a years worth of snapshots on the backup pool?  Whenever I run my `zfs recv` it deletes ALL my snapshots on the backup pool (except the two most recent which are on the live root ZFS pool).

How do other people handle their incremental ZFS backups? 

I want to keep up to a years worth of snapshots on the backup pool but I only want up to two snapshots on the root ZFS pool.

Can this be done? I've read the zfs man() page and tried so many `zfs recv` command switches that my head is spinning now.

Can anyone help?  :q


----------



## t1066 (Feb 4, 2014)

How about using `zfs diff` to see what has changed in the receiving end?


----------



## xy16644 (Feb 5, 2014)

t1066 said:
			
		

> How about using `zfs diff` to see what has changed in the receiving end?



So I ran another incremental and it failed:


```
(/support)$ zfs send -R -i zroot@a zroot@b | zfs receive -duv zstore/dailysnapshots
receiving incremental stream of zroot@b into zstore/dailysnapshots@b
snap zstore/dailysnapshots@b already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr@b into zstore/dailysnapshots/usr@b
cannot receive incremental stream: destination zstore/dailysnapshots/usr has been modified
since most recent snapshot
warning: cannot send 'zroot/usr@b': Broken pipe
```

I then ran `zfs diff zroot/usr@b zstore/dailysnapshots/usr@b` and it said:


```
cannot open '': invalid dataset name
Not an earlier snapshot from the same fs: invalid name
```


----------



## t1066 (Feb 6, 2014)

You should run
`zfs diff zstore/dailysnapshots/usr@b zstore/dailysnapshots/usr`
to see what had changed in zstore/dailysnapshots/usr


----------



## Savagedlight (Feb 6, 2014)

There's one thing I find confusing here. Why do you create a secondary structure called 'dailysnapshots'? With ZFS send/recv, you can transfer live file systems up to the latest snapshot without interruption. You don't have to copy stuff from the original source to some secondary structure to transfer. Unless those files are not stored on ZFS to begin with; although they seem to be.


----------

