# Can't write a simple rsync backup script



## nickednamed (Jun 21, 2013)

I'm trying to write a simple net/rsync backup shell script, but I'm having problems with wildcards and escape characters.

There is one line in my script which is causing all the problems, yet it works fine from the console. Essentially, I'm trying to copy my dotfiles and hidden folders from:
home/user/ to home/user/backups/dotfiles.

Here is the command: `# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles`

When used in a script, rsync seems to ignore the escape character and will include all my top level home directories, along with /home/user/backups/ in the source directory, which leads to another backup folder in my backup destination, recursively on, forever, i.e. I end up with: /home/user/backups/dotfiles/backups/dotfiles/backups/dotfiles/...

I've tried using quotes, full paths, and many variations of both, but the script will not behave as it does on the command line (many of the variants worked well from the command line). What am I missing here?


----------



## DutchDaemon (Jun 21, 2013)

Just use rsnapshot, which makes incremental hourly/daily/weekly/monthly backups using rsync, including rollbacks. It takes less time to set up than to write another variant. As to your nested directories, make sure you understand the essential difference to rsync between /some/directory and /some/directory/, especially when using rsync recursively.


----------



## wblock@ (Jun 21, 2013)

sysutils/rsnapshot is not always a replacement for using `rsync` directly.  There are assumptions built in to the way it does things that may not be what the user wants to achieve.

Anyway:


> ```
> rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.[color="Red"]*[/color] ~/backups/dotfiles
> ```



Untested, but the dot is escaped and the star is not.  The dot is not a special character to the shell, but it will try to expand the star before running the command.  That may be done differently depending on the interactive shell versus the interpreter shell.


----------



## NewGuy (Jun 21, 2013)

I haven't tested this theory, but I wonder if rsync is respecting symbolic links? I'm assuming your home directory is really under the /usr directory and not under /home. If the script is expanding "~" as /home then rsync might be handling the symlink in unexpected ways. Have you tried putting in the (real) full path to your home directory?


----------



## jalla (Jun 21, 2013)

nickednamed said:
			
		

> Here is the command: `# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles`
> 
> When used in a script, rsync seems to ignore the escape character and will include all my top level home directories, along with /home/user/backups/ in the source directory, which leads to another backup folder in my backup destination, recursively on, forever, i.e. I end up with: /home/user/backups/dotfiles/backups/dotfiles/backups/dotfiles/...



Your problem is the use of '*~/.**' as it matches both the current directory and the parent ('.' and '..'). '*.??**' is usually an acceptable workaround.

PS. And I don't believe you when you say this works interactively


----------



## phoenix (Jun 23, 2013)

nickednamed said:
			
		

> I'm trying to write a simple net/rsync backup shell script, but I'm having problems with wildcards and escape characters.
> 
> There is one line in my script which is causing all the problems, yet it works fine from the console. Essentially, I'm trying to copy my dotfiles and hidden folders from:
> home/user/ to home/user/backups/dotfiles.
> ...



Skip doing it on the command-line, and use an --include-from= file, and list your patterns there.


----------



## nickednamed (Jun 23, 2013)

DutchDaemon said:
			
		

> Just use rsnapshot, which makes incremental hourly/daily/weekly/monthly backups using rsync, including rollbacks. It takes less time to set up than to write another variant. As to your nested directories, make sure you understand the essential difference to rsync between /some/directory and /some/directory/, especially when using rsync recursively.



Thanks. I had read the part of the manual about trailing slashes, and experimented accordingly. But I'll re-read that part of the manual for good measure.


----------



## nickednamed (Jun 23, 2013)

jalla said:
			
		

> Your problem is the use of '*~/.**' as it matches both the current directory and the parent ('.' and '..').
> 
> '*.??**' is usually an acceptable workaround.
> 
> PS. And I don't believe you when you say this works interactively



Thanks, I'll try to remember .??* in the future. I'm off to see if it works. And yes, my command does work interactively (just tried it again to make sure! ). Is there some reason it shouldn't other than ~/.* matching the current directory and the parent ('.' and '..')?


----------



## nickednamed (Jun 23, 2013)

phoenix said:
			
		

> Skip doing it on the command-line, and use an -- include-from= file, and list your patterns there.



Thanks. That's a much cleaner way of doing it! ) I suppose it will be easier to maintain a number of different include files for different purposes too. I'll get cracking on one now!


----------



## nickednamed (Jun 23, 2013)

OK, so I found a solution (other than the ones already shown to me by others in this thread). I've changed:
`# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/\.* ~/backups/dotfiles`

Now it reads:
`# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/.[^.]* ~/backups/dotfiles`

It works fine now. I'll go back and try a few of the other solutions too. Thanks for the help.

EDIT:

The following also works, both from the command line, and from within the script:
`# rsync -avzu --delete --delete-excluded --exclude-from=/home/user/.rsync.exclude ~/.??* ~/backups/dotfiles`


----------



## jalla (Jun 23, 2013)

nickednamed said:
			
		

> Thanks, I'll try to remember .??* in the future. I'm off to see if it works. And yes, my command does work interactively (just tried it again to make sure! ). Is there some reason it shouldn't other than ~/.* matching the current directory and the parent ('.' and '..')?



Stick to your own solution ('.[^.]*'). It's better than my quick-and-dirty workaround.

With regard to the '.*' I have a hard time seeing how this could possibly work. You can do an `ls .*` to verify that this actually expands to all files both in the current directory and the parent.


----------



## nickednamed (Jun 27, 2013)

I'm having a few problems with the include and exclude files...

Most of them are working well, but when I try to exclude ALL files except for those in the include list, nothing gets included (I have used full paths throughout).

Here is what I'm trying to do - I've made an include file which lists a bunch of important system files, such as etc/rc.conf and boot/loader.conf which happen to be dotted around the file system, hence the source directory is "/". 

Because of this, I want to exclude ALL directories and files, except for the ones listed in the include file:


```
rsync -avzu --delete --delete-excluded --exclude='*' --include-from=/usr/home/user/.rsync/system.files /* ~/backups/temp
```

Putting the "exclude" after the "include" has no effect:


```
rsync -avzu --delete --delete-excluded --include-from=/usr/home/user/.rsync/system.files --exclude='*' /* ~/backups/temp
```

So how do I exclude all files by default, apart from the ones listed in my include file?


----------



## phoenix (Jun 27, 2013)

Rsync uses a first-match-wins filter rule set.  */** is an implicit include rule for everything under */*

Instead of using */** and explicit *--include* and *--exclude* files, you should use *--filter* and create a proper rule list.  See the FILTER RULES section of the rsync(8) man page for details.  Create a single file that lists the explicit include/exclude rules.  And only use */* as the source directory on the command-line.


----------

