# help with rsync/unrar script



## wonslung (Mar 2, 2010)

I am trying to come up with a scripted solution to this:


I have a server which functions as a seedbox.  I've got it working perfectly and i've set it to move finished downloads to a COMPLETE/ folder, that way i can easily rsync the dir from the remote server to my local box.  This is all well and good but, now i have a new problem i'd like to solve.

If i use rsync, i have no problem RETRIEVING the files, but i have to leave the local copy of the downloaded files on my box until i decide to stop seeding them (because if i remove it locally, it will just download again as soon as the next rsync is run)

SOOO what i'm looking to do is this:  somehow keep a history of files which i have already sync'd and only download the ones which do not match this history, while at the same time removing items from this history file when they no longer exist on the remote server.  After this, i'd also like to be able to take these newly synced files, unrar them and delete everything except the rar'd copy without having to worry about it all downloading and unraring again.


If anyone has ANY idea of where a good place to start with this is, please let me know.

(i reccon the rsync issue should be tackled first but i'm kind of stuck)


----------



## rusty (Mar 2, 2010)

This unpacking script is on a forums I visit. I've not used it myself so I can't vouch for it in anyway but hopefully it's useful.


```
descene â€” Mass RAR unpacker

[color="Red"]v1.1[/color] (2010-02-21) [url]https://pastee.org/2p6mz[/url]
 - Proper handling of *.partXX.rar files
 - Handle up to 1000 RAR parts (foo.r999)
 - Ignores are now case-insensitive

[color="Red"]v1.0[/color] (2010-02-21) [url]https://pastee.org/vr6ut[/url]
 - Initial version


This script unpacks RARed releases and also copies or moves plain movie files, or
directories containing them, to a destination directory.
[CODE]
THIS SCRIPT COMES WITH NO WARRANTY. IF IT EATS YOUR FILES, THAT IS ENTIRELY
YOUR PROBLEM.

EXAMPLES
   # copy to a destination (add "-m" to move)
   descene -d ~/Videos/Movies ~/bt/completed/Blockbuster.XviD.AC3-NoGroup/

   # unpack in-place and keep the archive files
   descene -d. Blockbuster.XviD.AC3-NoGroup

   # unpack in-place and remove the archive files
   descene -md. Blockbuster.XviD.AC3-NoGroup

NOTES
 - save the script as "descene" and make it executable (chmod a+x descene)
 - you need to have Python >= 2.5 installed (aptitude install python-minimal)
 - "unrar" has to be available on your path
 - should work on any *nix and Windows, but only tested on Ubuntu
 - on Linux, use "easy_install --prefix /usr/local trash-cli" when you want
   files to be trashed rather have them instantly deleted
 - use copy mode (no -m) until you feel comfortable with the options and
   things work like you wanted them to; also, use -v to see details of what
   happens


Usage: descene [options] <pathnames>...

Options:
 --version                             show program's version number and exit
 -h, --help                            show this help message and exit
 -q, --quiet                           omit informational logging
 -v, --verbose                         increase informational logging
 -n, --dry-run                         don't do anything, only show what would've been done
 --fix                                 edit this script
 -d DESTDIR, --destination=DESTDIR     destination directory, use '.' for in-place operation
 -m, --move                            move instead of copying
```
[/code]


----------



## wonslung (Mar 3, 2010)

rusty said:
			
		

> This unpacking script is on a forums I visit. I've not used it myself so I can't vouch for it in anyway but hopefully it's useful.
> 
> 
> ```
> ...



that's cool, but i'm still stuck with the rsync issue.

my problem, is:

I want to rsync data from a remote server to a local server but i only want to copy the date one time.  After it's been copied, i want it to ignore the same data if it rsyncs a second time because it won't be on the local machine anymore

currently, i can get rsync working but i can't figure out how to make it keep track of what it has already downloaded and avoid redownloading ti again, unless i leave all the data on the local machine untouched....but i don't want to keep 500 gb of data on my local machine just to keep from redownloading it over and over.


----------



## DutchDaemon (Mar 3, 2010)

rsync compares the files you *have*, not the files you *had* ... That is simply not a function of rsync; it has no recollection, history, or been-there-done-that list. If you want something like that, you'll have to write some sort of history function.


----------



## wonslung (Mar 3, 2010)

DutchDaemon said:
			
		

> rsync compares the files you *have*, not the files you *had* ... That is simply not a function of rsync; it has no recollection, history, or been-there-done-that list. If you want something like that, you'll have to write some sort of history function.



I know that, that's the problem.

It should be possible to script it with the filter/exclude rules


----------



## DutchDaemon (Mar 3, 2010)

Maybe a combination of 'touch', 'find' and 'rsync' would work. Every time you rsync, touch a file on the source system (somewhere in /var/tmp, for example), and use 'find files in source directory that are newer than the previously touched file' to rsync those specific files over (and then touch that file again to update the timestamp). This would work just as well with scp, because you're not really synchronising anything anymore, just copying over 'files newer than a  certain timestamp'.


----------



## wonslung (Mar 4, 2010)

actually, that's a good idea.

I never thought of using scp for this....i have no idea exactly how to make it work...but thanks for the idea


so you're saying i could do something like.....create a database using touch?  so use find to get the new names, then scp them one at a time, use touch at the same time with a pipe to create a touched file of the same name...i get that part but how do i make it use those files to ignore them the next time...that's what i'm confused on...


----------



## jalla (Mar 4, 2010)

You may be overcomplicating the problem. Basically, what DD suggests is something like this


```
#!/bin/sh
for f in `find SRC -newer .timestamp` 
do
scp $f DST
done
touch .timestamp
```


----------



## jalla (Mar 4, 2010)

Or using a different approach, you could keep a history of files copied


```
#!/bin/sh
for f in `find SRC`
do
grep -qx $f .history || ( scp $f DST; echo $f  >> .history )
done
```


----------



## DutchDaemon (Mar 4, 2010)

Yep, something like that. Just 'touch' a file right after copying over your files. It can be called anything, but .timestamp sounds functional. Now, all you do is tell 'find' to locate files that are _newer than the .timestamp file_, which should give you all the files that were created since .timestamp was last updated ... which was after you last copied files. The .timestamp file itself has no content, it's just an emtpy file with one basic function: it holds the timestamp of your last copy process to assist 'find' with finding newer files.


----------



## DutchDaemon (Mar 4, 2010)

Using a history would be possible, but you're opening a can of worms with odd filenames, spaces, non-ASCII characters, etc. You may find it difficult to get a proper match without using additional special regexps, escaped strings, quoted variables, etc. I think using a timestamped reference file is more KISS.


----------



## wonslung (Mar 5, 2010)

no, the timestamp idea is perfect i think.

I will work on that....thanks so much.

edit:

ok, i have it working but in the oposite direction to what i'd like....

basically...when i rsync now, i run it locally TO the seedbox...this is prefered because it doesn't require me to punch a hole in my firewall. 

the way i have it working now, i have to scp TO my local machine like this:

scp somefile wonslung@my.local.address:/somedir

i'd rather run it like this:

scp wonslung@my.remote.address:/somedir/somefile somedir/

know what i mean?

The problem is, i can't figure out how to do "find" over ssh correctly....

i've run other commands over ssh in this way before but this isnt' working the way i'd expect.


EDIT2:

damn, i feel dumb....i figured it out.


```
ssh wonslung@seedbox.example.com 'find /home/wonslung/test/ -newer /home/wonslung/test/.timestamp'
```


----------



## wonslung (Mar 5, 2010)

ok, i got it working but i'm a little confused as to how to get the variables to work the way i want

anyways, this is what works:


```
#! /bin/sh
RMT="wonslung@seedbox.example.com"
SRC="/home/wonslung/test/"
DST="/export/home/wonslung/test/"
TMST="'$SRC'/.timestamp"

for f in `ssh $RMT 'find /home/wonslung/test/ -newer /home/wonslung/test/.timestamp`
do
scp -r "$RMT":$f $DST
done
ssh $RMT 'touch /home/wonslung/test/.timestamp'
```


this doesn't work: 

```
#! /bin/sh
RMT="wonslung@seedbox.example.com"
SRC="/home/wonslung/test/"
DST="/export/home/wonslung/test/"
TMST="'$SRC'/.timestamp"

for f in `ssh $RMT 'find $SRC -newer $TMST`
do
scp -r "$RMT":$f $DST
done
ssh $RMT 'touch $TMST'
```


i'm sure it has something to do with the the fact i'm running it remotely, is there anyways to pass variables in such a way?


----------



## wonslung (Mar 5, 2010)

another question i have is this:

Will this have any problem with unfinished files?

let's say a file is being moved to the dir i'm running the find on and it isn't done yet, will it try to copy the file?  or does the timestamp of the file get updated AFTER the move?

hrm...this doesn't seem to be working recursively....

I must have done something wrong,....

Edit:

I must be doimng something wrong with my variables....when i do it one way, it ends up scp'ing everything , when i do it another, it doesn't work it all....if i just put the commands in directly they work....so until i figure out why my variables fdon't wokr the way i think they should i went with this

```
#! /bin/sh
RMT="wonslung@seedbox.example.com"
SRC="/home/wonslung/Complete/"
DST="/tank/nas/dump/torrents/seedbox/"
TMST="'$SRC'/.timestamp"

for f in `ssh wonslung@seedbox.example.com 'find /home/wonslung/Complete/ -newer /home/wonslung/Complete/.timestamp`
do
scp -r wonslung@seedbox.example.com:"$f" /tank/nas/dump/torrents/seedbox/
done
ssh $RMT 'touch /home/wonslung/Complete/.timestamp'
```

of course my variables don't do anything in this script but it works...i'd like to figure out how to use the variables correctly though.


----------



## wonslung (Mar 5, 2010)

Also, now it is working....somewhat, but in the situation where i have a couple new dirs filled with files, i end up with it copying the dir with all the files and then copying the files inside the dir as well...twice....

like this:

```
wonslung@wonslung-raidz:~$ cd /tank/nas/dump/torrents/seedbox/
wonslung@wonslung-raidz:/tank/nas/dump/torrents/seedbox$ ls
obj-tosh.0.s02e07.nfo           obj-tosh.0.s02e08.r00
obj-tosh.0.s02e07.r00           obj-tosh.0.s02e08.r01
obj-tosh.0.s02e07.r01           obj-tosh.0.s02e08.r02
obj-tosh.0.s02e07.r02           obj-tosh.0.s02e08.r03
obj-tosh.0.s02e07.r03           obj-tosh.0.s02e08.r04
obj-tosh.0.s02e07.r04           obj-tosh.0.s02e08.r05
obj-tosh.0.s02e07.r05           obj-tosh.0.s02e08.r06
obj-tosh.0.s02e07.r06           obj-tosh.0.s02e08.r07
obj-tosh.0.s02e07.r07           obj-tosh.0.s02e08.r08
obj-tosh.0.s02e07.r08           obj-tosh.0.s02e08.r09
obj-tosh.0.s02e07.r09           obj-tosh.0.s02e08.r10
obj-tosh.0.s02e07.r10           obj-tosh.0.s02e08.r11
obj-tosh.0.s02e07.r11           obj-tosh.0.s02e08.rar
obj-tosh.0.s02e07.rar           obj-tosh.0.s02e08.sfv
obj-tosh.0.s02e07.sfv           Tosh.0.S02E07.HDTV.XviD-OBjECT
obj-tosh.0.s02e08.nfo           Tosh.0.S02E08.HDTV.XviD-OBjECT
wonslung@wonslung-raidz:/tank/nas/dump/torrents/seedbox$ cd Tosh.0.S02E07.HDTV.XviD-OBjECT/
wonslung@wonslung-raidz:/tank/nas/dump/torrents/seedbox/Tosh.0.S02E07.HDTV.XviD-OBjECT$ ls
obj-tosh.0.s02e07.nfo  obj-tosh.0.s02e07.r04  obj-tosh.0.s02e07.r09
obj-tosh.0.s02e07.r00  obj-tosh.0.s02e07.r05  obj-tosh.0.s02e07.r10
obj-tosh.0.s02e07.r01  obj-tosh.0.s02e07.r06  obj-tosh.0.s02e07.r11
obj-tosh.0.s02e07.r02  obj-tosh.0.s02e07.r07  obj-tosh.0.s02e07.rar
obj-tosh.0.s02e07.r03  obj-tosh.0.s02e07.r08  obj-tosh.0.s02e07.sfv
wonslung@wonslung-raidz:/tank/nas/dump/torrents/seedbox/Tosh.0.S02E07.HDTV.XviD-OBjECT$
```

any ideas how to fix this?

i can remove the -r from scp but then i end up with just a bunch of files without the dirs...i GUESS this is ok but it isn't exactly what i'm looking for.....


Edit:

I can get scp for dir's to work like this.....i'm wondering if this is going to cause problems though...dont' know enough about how timestamps work to predict


```
#!/bin/sh

for f in `ssh wonslung@seedbox.example.com 'find /home/wonslung/Complete/ -type d -newer /home/wonslung/Complete/.timestamp`
do
scp -r wonslung@seedbox.example.com:"$f" /tank/nas/dump/torrents/seedbox/
done
ssh wonslung@seedbox.example.com 'touch /home/wonslung/Complete/.timestamp'
```

my main worry is, if a dir inside of another dir is added, does it update the timestamp on the lower level dir?  if so is this going to cause me to scp the entire contents of the dir?  if so i'm no better off than where i started.

also, this doesn't work for files added which aren't part of a dir....this isn't very often the case but i'd like to figure a way to make those files work too....


----------



## wonslung (Mar 5, 2010)

yah, nether of these work because i have a recursive dir structure.  As soon as a new dir is copied into /home/wonslung/Complete/TV/

it tries to scp the entire structure......is there any way around this?

I guess i could use this method on the lowest level......

but then i'll need to use a different script for each subdir in /home/wonslung/Complete/

(one for /home/wonslung/Complete/TV/, one for /home/wonslung/Complete/Movies/ one for /home/wonslung/Complete/Music)

any ideas on this?


----------



## DutchDaemon (Mar 5, 2010)

wonslung said:
			
		

> this doesn't work:
> 
> ```
> #! /bin/sh
> ...



You're passing the variables in single quotes, which will make them literal. Try using double quotes for the remote command.


```
# SRC=something
# echo '$SRC' and "$SRC"
$SRC and something
```


----------



## wonslung (Mar 5, 2010)

ahh, yeah, i forgot about that.

So this should work:


```
#! /bin/sh
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp

for f in `ssh $RMT "find $SRC -type d -newer $TMST"`
do
scp -r $RMT:$f $DST
done
ssh $RMT "touch $TMST"
```

is this going to cause me any issues thouhg? i need to test i guess..i don't want it to try to scp the entire dir of /home/wonslung/Complete/TV just because a new file ends up in there

edit:

just tested it, has the same problem....if a new dir is copied into /home/wonslung/Complete/TV/ it ends up updating the timestamp on /TV and i download the entire thing.

changing it to

```
#! /bin/sh
RMT=wonslung@seedbox.wonslung.com
SRC=/home/wonslung/Complete/TV/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp

for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST"`
do
scp -r $RMT:$f $DST
done
ssh $RMT "touch $TMST"
```

works...but it's not exactly what i was hoping for.  This method means i'll end up needing to script like this for each dir....and i'm not sure if there are going to be other unforseen issues.


----------



## wonslung (Mar 5, 2010)

Also, i've got 2 other issues i need to figure out.

One, i need to figure out how to add multiple finds and scp's to the same dir (one for files and one for directories)

because every so often a single file will end up in /home/wonslung/Complete/TV/ that needs to be scp'd as well.  I've got the working command i think

```
for g in `ssh $RMT "find "$SRC"* -type f -maxdepth 0 -newer $TMST"`
```
(or is it -maxdepth 1?)  eitherway, i'm sure it's one of those....i want it to look for single files in /home/wonslung/Complete/TV/ but not in /home/wonslung/Complete/TV/Someotherdir/


but, what is the best way to add it to the script?  should i pipe it behind the first one or should it be after the first done?

```
#! /bin/sh
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp

for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST"`|for g in `ssh $RMT "find "$SRC"* -type f -maxdepth 0 -newer $TMST"`
do
scp -r $RMT:$f $DST
scp $RMT:$g $DST
done
ssh $RMT "touch $TMST"
```

or

```
#! /bin/sh
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp

for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST"`
do
scp -r $RMT:$f $DST
for g in `ssh $RMT "find "$SRC"* -type f -maxdepth 0 -newer $TMST"`
scp $RMT:$g $DST
done
ssh $RMT "touch $TMST"
```

and the second issu, i need to make sure this script won't run if it's already running.  I've handled this in the past with pgrep...i guess i could use the same idea for this.


----------



## DutchDaemon (Mar 5, 2010)

```
touch .lockfile
rm .lockfile
```


----------



## wonslung (Mar 5, 2010)

ok, i got the 2 find commands working....i did it like this:


```
#! /bin/sh
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp

for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST|find "$SRC"* -type f -maxdepth 0 -newer $TMST"`
do
scp -r $RMT:$f $DST
done
ssh $RMT "touch $TMST"
```

i'm not sure exactly how you mean to use the lockfile.....i used something like this before:

```
if pgrep -u $USER $SERVICE
then
```
I'm sure it's somethign similar....but i'll have to see if i can read up on it

or do i use "while"


edit:

I figured it out


```
LOCKFILE=/home/wonslung/.lockfile
if [ -f $LOCKFILE ]
then
echo "lockfile, exiting"
exit 0
else
touch $LOCKFILE
fi
```


----------



## DutchDaemon (Mar 5, 2010)

well done 

Don't forget to remove it again somwehere in the process, though.


----------



## wonslung (Mar 5, 2010)

I'm making progress =) thanks so much for helping me with this...

```
#! /bin/bash
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
touch $LOCKFILE 
fi    
for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST|find "$SRC"* -type f -maxdepth 0 -newer $TMST"`
  do
scp -r $RMT:$f $DST
done

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```

I have another question, if i needed to add more to a single line..but i didn't want it going on for so long, how do i break it into other lines...do i use \
?


----------



## DutchDaemon (Mar 5, 2010)

Yep, end line with \
and continue on next line.


```
cd /usr/src && \
make cleanworld && make cleandir && \
make -j 4 buildworld && \
(etc.)
```


----------



## wonslung (Mar 5, 2010)

ok, question.....why doesn't this work?


I know it's something to do with the quotes....but i can't pass all 4 commands....2 work fine though.

```
#! /bin/bash
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
touch $LOCKFILE 
fi    
for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST|find "$SRC"* -type f -maxdepth 0 -newer $TMST|find "$SRC2"* -type d -newer $TMST|find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
  do
scp -r $RMT:$f $DST
done

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------



## wonslung (Mar 5, 2010)

changing the pipes to && made it work.


----------



## DutchDaemon (Mar 5, 2010)

Separate them with ";" or "&&" perhaps? I don't understand all the pipes.

Or replace the quoted variables with ${SRC} and ${SRC2}.


----------



## wonslung (Mar 5, 2010)

ok, this is the finished (for now, until i can make unrar work) script


```
#! /bin/bash
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```

next thing i need to do is make it unrar files to a new dir, and delete the rar files/dirs

Thanks so much DD, your help has been invaluable.


----------



## wonslung (Mar 7, 2010)

so now i need to add the unrar part of the script.  currently, this script is doing pretty much what i want when it comes to downloading....so heres what i want to do next.

I need to be able to recursively unrar stuff, and dump what comes out to a single directory. I understand how to do this, but where i run into problems is when i end up with split rar files all named .rar so what i need to do is somehow ignore all but one of those OR make unrar skip unraring if the file already exists....i'd RATHER do the first option if possible...

Also, i'd like to be able to delete the rar files when they are done being extracted, and all split rars' with them...basically, the entire dir they are in..but i don't want to accidently delete stuff that hasn't been processed....so....this is going to be harder than the other part....



When i'm done though, i should have a script which will auto-download everything (via cron) from my seedbox, unrar it all into a dir which can be processed for sorting later.


----------



## wonslung (Mar 7, 2010)

ok, so i figured out how to find rar files and unrar them into thier dir's...what i want to do now is, if the unrar is successful (how do i tell this?)  i want to delete all the .rar and .r[\d\d] files and move the directory containing the unrarred file.

The problem i'm having is how do i know if it's successful.....anyways, this is what i have so far..



```
for d in `$FIND	"$DST"*	-type d`
   do
      for r in `$FIND $d -name "*.rar*"`
	 do
            unrar e -r -inul $r $d
      done   
done
```


the $FIND part is set to my find command. $DST is the same from earlier in the thread.

thanks


----------



## wonslung (Mar 7, 2010)

i'm wondring....this seems to work:



```
for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done
```


edit 2

i'm too tired...i see what i did wrong....fixed it

(i forgot to add the FIND= line"


does anyone see any issues with this?

edit:

ok, this is odd...when i put the two scripts together i get this error:

```
./seedbox.sh: line 23: /tank/nas/dump/torrents/seedbox/Caprica.S01E06.720p.HDTV.x264-CTU: cannot execute [Is a directory]
```


but they work fine in two separate scripts....why is this?

here is the entire script which isn't working

```
#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done


rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------



## wonslung (Mar 7, 2010)

ok, i figured it out, i forgot to set the FIND vairable.....really dumb...anyways, here is my completed script which seems to work....if anyone sees any obvious issues let me know.


```
#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in `ssh $RMT "find "$SRC"* -type d -newer $TMST && \
        find "$SRC"* -type f -maxdepth 0 -newer $TMST && \
        find "$SRC2"* -type d -newer $TMST && \
        find "$SRC2"* -type f -maxdepth 0 -newer $TMST"`
     do
 scp -r $RMT:$f $DST
done

for d in `$FIND "$DST"* -type d`
   do
      for r in `$FIND $d -name "*.rar*"`
         do
            unrar e -r  $r $d;rm "$d"/*.r??; mv $d "$DST"Done
      done   
done


rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------



## wonslung (Mar 7, 2010)

ok, i have a problem.....this script fails on files with spaces in the names.....any idea how to fix that?


----------



## sixtydoses (Mar 7, 2010)

Try to add this in your script.


```
IFS='
'
```


----------



## wonslung (Mar 7, 2010)

sixtydoses said:
			
		

> Try to add this in your script.
> 
> 
> ```
> ...



i have no idea what this means....

edit:

Just read up on IFS, will try.
not sure where to add that to though


----------



## sixtydoses (Mar 7, 2010)

Add it before you do the copying. Sorry, I can't make your script to work, but I've faced this white space problem before.

Example:

```
[od@meh ~/temp]$ ls -l
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:51 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:52 white space dir

[od@meh ~/temp]$ srcf=`ls`
[od@meh ~/temp]$ echo $srcf
white space white space dir

[od@meh ~/temp]$ cp -vR $srcf ../dst/
cp: dir: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory
cp: space: No such file or directory
cp: white: No such file or directory

[od@meh ~/temp]$ IFS='
> '

[od@meh ~/temp]$ cp -vR $srcf ../dst/
white space dir -> ../dst/white space dir
white space -> ../dst/white space

[od@meh ~/temp]$ ls -l  ../dst
total 2
-rw-r--r--  1 od  od    0 Mar  7 21:55 white space
drwxr-xr-x  2 od  od  512 Mar  7 21:55 white space dir
```


----------



## wonslung (Mar 7, 2010)

sixtydoses said:
			
		

> Add it before you do the copying. Sorry, I can't make your script to work, but I've faced this white space problem before.
> 
> Example:
> 
> ...



adding that works for spaces but not " breaks it.

Still, this is progress


----------



## DutchDaemon (Mar 7, 2010)

Use double quotes around variables to include whitespace literally. 

So not $FILE, but "${FILE}". The curly braces prevent interaction with adjacent characters which may make the script not recognise a variable when it's called (e.g. PATH=/usr/ ; cd $PATHsrc -- this should be ${PATH}src)


----------



## wonslung (Mar 8, 2010)

DutchDaemon said:
			
		

> Use double quotes around variables to include whitespace literally.
> 
> So not $FILE, but "${FILE}". The curly braces prevent interaction with adjacent characters which may make the script not recognise a variable when it's called (e.g. PATH=/usr/ ; cd $PATHsrc -- this should be ${PATH}src)



I've been reading up on this issue and it seems to be caused by the ` ` quotes....i have no idea how to fix this issue.

I will try your modifications too but see this: and tell me what you think.


http://mywiki.wooledge.org/BashPitfalls#for_i_in_.60ls_.2A.mp3.60


I have no idea how to rewrite my script without this.


edit:  I made some modifications....the whitespace thing is solved but it's solved by a method which breaks other stuff....the problem is two fold.

First, ssh and scp have a whitespace problem ....i had to change this:


```
scp -r ${RMT}:$f ${DST}
```
to this:

```
scp -r ${RMT}:"\"$f\"" ${DST}
```


to make it get paste the whitespace issue....but that breaks stuff with "'s in it.....i think the ultimate issue is that using the `'s are bad.
I tried to fix that by using this:    $(  )  but it didn't help much



anyways, this is what i have so far but i think i'm going to have to re-write everything to get it to work flawlessly....luckily it works for 99% of my files...so i have itme




```
#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    
     for f in $(ssh $RMT "find ${SRC}* -type d -newer $TMST && \
        find ${SRC}* -type f -maxdepth 0 -newer $TMST && \
        find ${SRC2}* -type d -newer $TMST && \
        find ${SRC2}* -type f -maxdepth 0 -newer $TMST")
     do
 scp -r ${RMT}:"\"$f\"" ${DST}
done

for d in $($FIND ${DST}* -type d)
   do
      for r in $($FIND $d -name "*.rar*")
         do
            unrar e -r  $r $d;rm ${d}/*.r??; mv ${d} ${DST}Done
      done   
done

for m in $($FIND ${DST} -maxdepth 1 -name "*.avi" -o -name "*.mkv")
   do
       mv $m ${DST}Done
done
rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------



## sixtydoses (Mar 8, 2010)

It is a good practice to define IFS when you need it, and reset it back when you're done with it.

e.g.:

```
# store original IFS and assign a different value to it
OLDIFS=$IFS
IFS='
'
 
# do whatever you want here
 
# reset it back
IFS=$OLDIFS
```


----------



## wonslung (Mar 8, 2010)

cool, doesn't help me with this issue though.

But, i do like learning about "best practices"  apparently my script is FULL of "horrid" issues...


----------



## sixtydoses (Mar 8, 2010)

wonslung said:
			
		

> to make it get paste the whitespace issue....but that breaks stuff with "'s in it.....i think the ultimate issue is that using the `'s are bad.
> I tried to fix that by using this:    $(  )  but it didn't help much



Sorry I don't really understand.. you were saying you're having problems transfering files "'s" in it? Something like, "some movie's.rar"?


----------



## wonslung (Mar 8, 2010)

sixtydoses said:
			
		

> Sorry I don't really understand.. you were saying you're having problems transfering files "'s" in it? Something like, "some movie's.rar"?





actually, i think single quotes (apostrpe's) work..it's double quotes which break it.  Theres soemthing else which breaks it as well but it's pretty minor.

one of the biggest problems is that scp itself doesn't handle whitespace, so to make the whitespace work, i have to do this:




`scp -r remoteserver:"\"file name with whilespace\"" Destination`


This fix breaks sending quotes i think...


----------



## LiMPiNg (Mar 12, 2010)

First, I would like to thank you wonslung for all the work you have put in this script already. I was looking for something similar to use on my FreeNAS server, without the copying from the seedbox - I just want it to automatically extract files from my download directory to another directory of extracted files. Being a FreeBSD/UNIX rookie, I stumbled my way through it and have it working how I want... almost!

What do you do for archives that have .partXX.rar filenames instead of .rXX? ie, xxxx.part01.rar, xxxx.part02.rar . I ran into such a torrent, which is not uncommon, and it threw the whole script off due to all the .rar files in the dir. I would love to figure this out on my own, but I know nothing about scripting and simply modified yours to fit my needs.

I know the following code is what needs to be worked on for this.

```
for r in $($FIND $d -name "*.rar*")
```

I am guessing having it do an if search for a part01.rar and extract that if it exists and then do an else to process the single .rar if there is no part01.rar is the key, but how to implement that exactly is beyond my limited scripting abilities. Any help would be appreciated.


----------



## wonslung (Mar 12, 2010)

LiMPiNg said:
			
		

> First, I would like to thank you wonslung for all the work you have put in this script already. I was looking for something similar to use on my FreeNAS server, without the copying from the seedbox - I just want it to automatically extract files from my download directory to another directory of extracted files. Being a FreeBSD/UNIX rookie, I stumbled my way through it and have it working how I want... almost!
> 
> What do you do for archives that have .partXX.rar filenames instead of .rXX? ie, xxxx.part01.rar, xxxx.part02.rar . I ran into such a torrent, which is not uncommon, and it threw the whole script off due to all the .rar files in the dir. I would love to figure this out on my own, but I know nothing about scripting and simply modified yours to fit my needs.
> 
> ...



dude, i'm pretty new to this too and this script is REALLY bad.


i'm also thinking of changing how i do unrar.....i'm thinking instead of using a "for" loop, i'll just use something more like:

find somedir -type f -name *.rar -exec unrar e {} \;

but...that's not quite right and needs some work....i'm very new to this too..


I've been thinking a lot about it and i'm actually thinking about changing my method....this is what i've been thinking of, if anyone has any ideas how i can do this let me know because i'm just in the "idea" phase.

The Problem:

As i've said before, this script breaks on specific file names.  I've been thinking about it and rsync is just a better tool, but the major issue is that it doesn't work for what i want and writing a script for an exclude file just isn't easy at all....so this is my idea:

somehow use symbolic links.

Write a script to create symbolic links (should be easy with the same method i used for downloading)

Then have rsync download this directory...the problem is i need to then have rsync DELETE the link when it's done.

any ideas?

edit:

another big problem i think i forgot to mention is:

on very large files, it messes up.

situation:  downloading a giant file, it takes some time, meanwhile some new stuff is added, the script doesn't pick it up.


----------



## wonslung (Mar 12, 2010)

This is what i'm about to test.

If this works the way i THINK it will, i'll be ok....i haven't put the unrar part in let, but i will

The reason i run the rsync command 3 times is i want to make sure i catch everything...tha'ts probably not foolproof....but i don't know what else to do....any ideas would be helpful.
nayways, this is what i'm ABOUT to test...


```
#! /bin/sh
LOCKFILE="/tmp/seedbox.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents/seedbox2/
TMST=$SRC.timestamp
LOG=/export/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG
rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG
rsync -aveL ssh ${RMT}:/home/wonslung/sync $DST >> $LOG


ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;
rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------



## wonslung (Mar 12, 2010)

ok, i had an error in it, but this is the script which is working:


this goes in, finds dirs which are new, makes a link for them in another dir, then rsync (following links) those dir

I set it to rsync 3 times, hoping that if something is added while the first rsync is going on, it will catch it....i'm trying to think of a better method for this..i'm thinking perhaps a double or tripple timestamp but i'm not entirely sure...must be a better way.


anyways, after that, it goes in and deletes the symbolic links.

heres the script, later i will add the logic to unrar and sort stuff

```
#! /bin/sh
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
LOG=/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG
rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG
rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```



EDIT:

I just realized the 3 rsync is stupid because it doesn't find any new files, i wasn't thinking....

so my question is:  other than just making it run everything again, is there any way to tell it to loop a couple times?

EDIT:

just found the answer.  a function.

heres editted script:

```
#! /bin/sh
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
LOG=/home/wonslung/log/rsync.log
IFS='
'
if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

link() {

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"
}

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```

Edit again:

okay, i'm SURE this has it's own issues which i'm not seeing yet.....but this is the best i can do so far.



```
#! /bin/bash
LOCKFILE="/tmp/.lockfile"
RMT=wonslung@seedbox.example.com
SRC=/home/wonslung/Complete/TV/
SRC2=/home/wonslung/Complete/Movies/
DST=/tank/nas/dump/torrents2/
TMST=$SRC.timestamp
FIND=/usr/gnu/bin/find
LOG=/home/wonslung/log/rsync.log
IFS='
'

if [ -f $LOCKFILE ]
        then
                echo "lockfile, exiting"
        exit 0
else
   touch $LOCKFILE 
fi    

link() {

ssh $RMT "find ${SRC}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"


ssh $RMT "find ${SRC2}* -type d -maxdepth 0 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"

ssh $RMT "find ${SRC2}* -type f -maxdepth 1 -newer $TMST \
-exec ln -s '{}' /home/wonslung/sync \;"
}

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG

link

rsync -Lave ssh ${RMT}:/home/wonslung/sync/ $DST >> $LOG


shopt -s extglob

for dir in ${DST}*/
  do
    cd "$dir" && $FIND . -type f -name '*.rar' \( ! -name 'part*.rar' -o -name 'part01.rar' \) -exec unrar e {} \;
done
for g in ${DST}*/
  do
    rm ${g}*.r?? ${g}*.sfv ${g}*.nfo
done
$FIND ${DST}* -maxdepth 0 -type d -exec mv {} /tank/nas/dump/Done \;




ssh $RMT "find /home/wonslung/sync -type l -exec rm '{}' \;"

rm -f $LOCKFILE ; ssh $RMT "touch $TMST"
```


----------

