# FTP cronjob : Upload from queue until full?



## Sokonomi (Jul 27, 2018)

Hello all,

I, as a total freeBSD beginner, will probably be completely in over my head for this one, but please try anyway. 

*What I want to achieve;*
The Laymen's explanation; I want my FreeNAS to upload new TV shows to my tablet every night, until the tablet is full.

I need a script to attempt uploading the top line of a queue file to a FTP destination and delete the top line if it succeeds, cycling until it gets a failure and then quitting. I would run it through a cronjob on my NAS.

Supposedly I have my TV show fetchers generate a list of new shows that looks like this;

/mnt/Tank1/TVshows/Battlebots s3e2.mkv
/mnt/Tank1/TVshows/Battlebots s3e3.mkv
/mnt/Tank1/TVshows/Battlebots s3e4.mkv

Can I make a script that works these into an FTP transfer?

Ive googled around and found an app called C-Kermit, supposedly this makes it easier?
If someone knows of an example script that handles queued FTP uploads in freeBSD I can dissect it and learn how it works. 

I hope you guys can help. If its too difficult ill have to resort to running DOS in a VM to do it with a batch file or something. 

Cheers,
- Soko


----------



## SirDice (Jul 27, 2018)

PC-BSD, FreeNAS, NAS4Free, and all other FreeBSD Derivatives


----------



## Sokonomi (Jul 27, 2018)

Yes, great, except this would run in a jail which is basically a flat freeBSD, is it not?
Plus I assumed ftp() is a base freeBSD function.


----------



## ShelLuser (Jul 27, 2018)

Sokonomi said:


> Yes, great, except this would run in a jail which is basically a flat freeBSD, is it not?


That's really not the point, it would have helped if you had actually read that URL; it isn't just your usual "_This is offtopic, please cease_" kind of answer. It is also about protection; both your system and our invested time.

I only know FreeBSD so I have no idea what changes have been made to FreeNAS. Don't forget: everything is customizable, so for all I know some binaries which are part of FreeBSD could have been removed on FreeNAS (to save space, or because it's not useful, etc, etc.). But it's also very likely that some things behave in a completely different way; making it very possible that certain answers to problems would easily work on FreeBSD but could actually do massive damage on FreeNAS (I'm not joking nor trying to make things look more gruesome just to make a point).

Another problem is that some derivatives are even based on unstable FreeBSD releases (developer snapshots) which use is actually discouraged by the FreeBSD community itself.

Most importantly: please note that SirDice never locked the thread nor forbid us from discussing.



Sokonomi said:


> Plus I assumed ftp() is a base freeBSD function.


There is no such thing as ftp() (as a function):

```
peter@zefiris:/home/peter $ man 4 ftp
No manual entry for ftp
```
But there _is_ something such as ftp(1), which I think is exactly what you need. If you read the manualpage you'll notice the -u option which allows you to upload file(s) to a remote destination.

So... maybe something like: `ftp -u upload@192.168.0.20 *` and then let it just sent everything over, it will complain (and quit) on its own once the remote no longer has any storage space.

To make this run every day you'd want crontab(1). First make a script which performs the upload:


```
#!/bin/sh
cd /mnt/tank1/TVShows
ftp -u upload@192.168.0.20 *
```
Then add a crontab entry which runs this script every day: `crontab -e`, for the right syntax to specify the time check crontab(5). Just be sure to specify the full name of your script. So, for example: /home/me/bin/upload-shows, crontab has a limited PATH.

This should be enough to get you started.


----------



## Sokonomi (Jul 27, 2018)

ftp(1) is just the actual transfer part though, which is easy enough and well documented.
What I need help with is processing the queue list, that stuff needs some hoop jumping I think.
Can you make a script read ftp exit codes?
C-Kermit apparently gives you a little more control over it, but im not sure how to handle the queue part.

The shows have been coming in faster than I can view them, and my tablets storage is limited,
so im having to deal with a bit of a backlog, thats why I cant just dump the entirety of /mnt/tank1/TVshows across.
If I can make a script FTP the files in like a feed, that would be most ideal.


----------



## ShelLuser (Jul 27, 2018)

Sokonomi said:


> What I need help with is processing the queue list, that stuff needs some hoop jumping I think.
> Can you make a script read ftp exit codes?


Ah yes, that's the part I somewhat overlooked.



Sokonomi said:


> Supposedly I have my TV show fetchers generate a list of new shows that looks like this;
> 
> /mnt/Tank1/TVshows/Battlebots s3e2.mkv
> /mnt/Tank1/TVshows/Battlebots s3e3.mkv
> ...


I don't know about you, but I know I can   Sorry for the wee bit of pun, weekend started and I'm in a good mood right now.

First: we're not going to be using any 3rd party stuff. The base system provides us with all the tools we need to make this happen. Second, because I'm also currently working on another hobby project of mine I decided to combine the two. So, what you're looking for based on functionality is roughly something like this:






For the record: Yes, I am geekish enough to enjoy UML. And no, I'm not trying to show off or anything; it's a hundred degrees out here which doesn't help with my ability to concentrate. As such a quick design like this _seriously_ helps me focus my thoughts. And since I had my design made anyway I figured I'd upload it to show off (but secretly!) 

Anyway, enough fun talk:


```
#!/bin/sh

# Queue file (use full path!)
queue=files.txt
# FTP host
host=breve
# FTP username
user=anonymous
# FTP password
pass=password
# FTP upload path
dir=/upload

## Don't change stuff beyond this line

count=0
size=`wc -l $queue | cut -d ' ' -f8`

for a in `cat $queue`; do
        check=$count;
        ftp -u ftp://$user:$pass@$host$dir/`basename $a` $a && count=$((count+1));
        if [ $check -eq $count ]; then
                echo "There was an error during the upload."    > /dev/stderr;
                break;
        fi
done

newsize=$((size - count))
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue
```
*Warning*: I expect you to test this for yourself as well before usage!  I did some proper testing and this works on my end, but as mentioned before I have 0 experience with FreeNas. 

So...  I think this speaks for itself. If you want to upload to the root directory then just leave dir empty. That should work but I haven't tested that. If it doesn't just let me know. Actually.. I'd appreciate it if you'd let me know anyway if this worked or not


----------



## Sokonomi (Jul 27, 2018)

Holy cow, I was expecting some examples so I could piece something together,
I didn't expect someone to do the whole thing for me.  Thanks!

That UML diagram is fantastic, I should have a look at that myself sometime.
Seems to be a great visual help for tracing how your code is gonna need to go.

As it appears, freeBSD script coding looks eerily familiar with other types of coding ive tangled with,
I can almost fully understand whats going on even though I've never written for freeBSD before.

So let me see if I interpreted it right;
It sets '*size*' to '*queue*' line count.
It runs a loop for every line in '*queue*'.
  It sets '*check*' to '*count*'.
  It adds +1 to '*count*' unless ftp encounters an error (full, no connection, login error, etc.).
  If '*check*' no longer equals '*count*' there was a failure, so it echoes error and exits the loop.
It subtracts 'count' from '*size*' and copies that number of lines from the tail end of '*queue*' over to a new file postfixed with '*_new*'.
It renames the *_new *file to the original *queue *file (I assume this overwrites it?).

I hope you dont mind me dissecting your coding,
I'm trying to learn from this as much as I can. 

I'm going to plop this into a test bed jail and run it, to see what comes out.
Thanks again, ill be back with results momentarily.

*EDIT*
I'm back, with some embarrassing results..
.. I cant get the script to run. 
I'm supposed to save it as a .sh file and run it as such, right?
I just get a mess of errors when I try executing it.





The base OS in this jail is 'FreeBSD 11.1-STABLE' so that should run, right?
The shebang line says it is supposed to be a .sh script, I presume?

I'm terribly sorry for being this ignorant with freeBSD.


----------



## ShelLuser (Jul 28, 2018)

Sokonomi said:


> As it appears, freeBSD script coding looks eerily familiar with other types of coding ive tangled with,


Note: this isn't specific "FreeBSD scripting" or coding, what you see here is a so called "Bourne shell script". At best you can call this "shell scripting", but even through we're using FreeBSD here that's actually not the important part. That's the shell itself (/bin/sh).



Sokonomi said:


> So let me see if I interpreted it right;
> It sets '*size*' to '*queue*' line count.
> It runs a loop for every line in '*queue*'.
> It sets '*check*' to '*count*'.
> ...


That's about right, yeah. As you can see the diagram only covered the rough (overall) functionality, I usually don't bother myself with details when working out designs.



Sokonomi said:


> I hope you dont mind me dissecting your coding,
> I'm trying to learn from this as much as I can.


Oh right, silly me, I forgot to license this code!   Naah, all good!



Sokonomi said:


> I'm back, with some embarrassing results..
> .. I cant get the script to run.
> I'm supposed to save it as a .sh file and run it as such, right?
> I just get a mess of errors when I try executing it.


Well, file extensions don't matter on FreeBSD, what does matter are permission flags. The easiest way to get this to work is to place the script somewhere, make it executable using `chmod +x test.sh` and then try running it using: `./test.sh` (this is assuming it's in the current directory).

However... `sh ./test.sh` should also work, with or without those flags. Making me question 2 things:

Is the script really still the same?
Is your shell (/bin/sh) actually the same Bourne shell?
Those error messages make little sense to me. So make sure there's no junk in the script, try: `less test.sh` to see what's actually in there and if that matches what I shared. Also: be sure to edit the variable values to match your own setup. I assume you already did so, but still wanted to mention it just in case.

You might also want to try: `echo $ENV` to see what it says, just in case.



Sokonomi said:


> I'm terribly sorry for being this ignorant with freeBSD.


No need, we all got to start somewhere and I knew up front what I got myself into. Trust me: I wouldn't be doing this if I had the idea that I was wasting some of my time


----------



## Sokonomi (Jul 28, 2018)

Wow, you're so friendly, its nice to be met with hospitality for a change! 
People have been pretty hostile for me asking 'stupid' questions on the freenas forums..

OH! I got it work now!
Very odd that it started working tough, all I did was exit the shell resave the script and try again.
Maybe it helped that I saved the script in notepad++ instead of regular notepad this time.
Now I'm actually getting feedback from ftp(1) in the terminal, so were getting somewhere.

However, It doesn't clear the last line off the queue file even though it did upload the file successfully,
so I think we might still be missing an EOF detection for exiting when the queue has been depleted entirely.

Maybe if we added

```
else
            echo "Queue depleted." > /dev/stdout;
            count=$((count+1))
            break;
fi
```
To the loop, would it fly?
Because if `count` equals `size` it means it went through all lines, right?

It also appears to be fussy about filenames that have spaces in them.
Ive tried encapsulating them in quotes but it doesn't seem to function that way.
So I guess the code is going to need some stuff to escape whitespaces?
Ive been banging on it all morning but I cant really figure out how to do this.
It seems to already be wrong when the line is captured in $a (its already missing part of the line there).

*EDIT*
After a whole lot of googling I can get it to escape the whole thing with `sed` but it still wont play nice with ftp.**

```
sed 's/ /\\ /g' files.txt > files2.txt
```
This seems to cough up the queue list with escaped whitespaces,
but the script wont properly use the whole line of this file either.
$a gets cut off too early for some reason; it always is just the first word, not the whole line.
I'm a little stumped on this one. :')

So lets say files.txt contains ..

```
/video file 1.mkv
/video file 2.mkv
/video file 3.mkv
```
.. how do we make that work?


----------



## ShelLuser (Jul 28, 2018)

Sokonomi said:


> OH! I got it work now!
> Very odd that it started working tough, all I did was exit the shell resave the script and try again.
> Maybe it helped that I saved the script in notepad++ instead of regular notepad this time.


Most definitely. That's the part I forgot earlier; dos2unix(1) is a thing as well, there's a difference in the way Windows / DOS or Unix saves their files.

And that leads me to: how is your queue file generated? Because that might also be an important factor here.



Sokonomi said:


> However, It doesn't clear the last line off the queue file even though it did upload the file successfully, so I think we might still be missing an EOF detection for exiting when the queue has been depleted entirely.


Hm, that shouldn't happen but it is possible that the problem is related to the format of the queue file, I'll do some testing here as well.

You did spot one most definite bug which I overlooked (told you I had concentration issues right now due to the extreme weather, looking back I even added bugs in the UML diagram  (which I left out of the script itself)):



Sokonomi said:


> It also appears to be fussy about filenames that have spaces in them. Ive tried encapsulating them in quotes but it doesn't seem to function that way.
> So I guess the code is going to need some stuff to escape whitespaces?
> 
> ..CUT..
> ...


So to address my helpful mood just this once: there is a reason for that. One of it being the fact that we're messing with a favorite Unix aspect of mine: shell scripting. But the most important aspect is showcased right here. Instead of just reporting bugs and/or asking for help you're also actively participating and trying to come up with reasons why things go wrong and try to solve 'm on your own. That is a really good motivator for me to lend a helping hand. Or keyboard in this case I suppose 

Anyway, the "spaces in filenames" is _definitely_ an oversight on my end. It takes quotes, but more than one due to the loop we're running. In fact, I sometimes even rely on /bin/csh when it comes to processing such files because it has a bit better support for that. I'll avoid that for now but do me a favor and run: `ls /bin/csh` to see if that shell also exists on your system. If it does then I know that I can always resort to "plan B" if we need to (I doubt it).

Anyway, give me a few hours. Need to pick up some groceries first and then I'll address the "spaces in filenames" problem and see if I can find a cause for the queue file not to get processed correctly.


----------



## Sokonomi (Jul 28, 2018)

It might have been notepad saving under a different charset or some other oddball thing, but notepad++ seems to know shell.  That little app is gods gift to coders.

As to where the queue is going to be coming from, I guess that should have been step 1. I'm hoping I can get the data out of sonarr via a custom script. If I can get another shell script (yay!) to append any new download to the tail end of the queue, I should be good to go. It shows *here* what variables you can extract from it, and it looks like an easy fish. Whenever the script triggers, it simply needs to slap `sonarr_episodefile_path` to the end of files.txt and shed be good to go. There's some snakes in the grass such as what happens when sonarr writes to it while the FTP shell script is actively processing the queue, but nothing too difficult to deal with I think.

As for the variables remaining up, this only seems to happen when I keep the shell open and run the script twice. They don't seem to flush once the script ends. Not sure if that would affect a cronjob, but its something to keep in mind.

And I love making things tick, automation is a passion for me, so writing shell scripts is basically a new toy.  I do have a weird way of learning though. I tend to look at realworld examples to dissect and pick apart rather than digging around in textbooks and manuals first. Id rather disassemble an engine to see what makes it go, than to just look at diagrams all day. This kind of behavior apparently irks a lot of people, or they misunderstand how I go about things. I wouldn't ask people to do my dirty work, but I do love it when they show me something that I can retool and adapt, because I keep looking up the parts they used to do it.

csh is present in my bin, so we got plan B if we need it.

`sed` looks to be capable of escaping whitespaces, having it parse the queue to a new file seems to work nicely. So that's probably the thing I should be using?

I'm also having trouble figuring out where its getting $a from. Where/how does that variable get assigned? The lines need to get quoted, but since that's not happening, only the first string 'sticks' to $a. Ive tried manually slapping some quotes on the lines in the queue file itself, but that doesn't seem to work. I think `cat` might be blocking the quotes to make it safe.


----------



## Sokonomi (Jul 29, 2018)

Ok, I think ive almost cracked it, please verify;

```
#! /bin/sh

# Queue file (use full path!)
queue=files.txt
# FTP host
host=192.168.178.22:2121
# FTP username
user=sokonomi
# FTP password
pass=8520Derp
# FTP upload path
dir=/Removable/MicroSD

## Don't change stuff beyond this line
IFS=$'\n'
count=0
size=$((`wc -l $queue | cut -d ' ' -f8`+1))

for a in `cat $queue`; do
    check=0
       ftp -u ftp://$user:$pass@$host$dir/`basename $a` $a && check=1 && count=$((count+1))
    if [ ! $check ]
    then # Upload failed.
        echo "There was an error during the upload." > /dev/stderr
        break
    elif [ $count = $size ]
    then # End of queue reached.
        echo "Queue depleted." > /dev/stdout
        break
    else
        echo "Succeeded, moving on to next line." > /dev/stdout
    fi
done

newsize=$((size - count))
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue
```

Ive fixed the indexing problem, but Im hung up on how it registers an `ftp` failure.
It seems to set the `check` flag regardless of the `ftp` command returning 0.
The use of `&&` is supposed to prevent that, right?
Any ideas?


----------



## ShelLuser (Jul 29, 2018)

Well, later than I anticipated (also had some IRL to sort out) but alas.

Plan B it is because /bin/sh isn't exactly the best shell when it comes to processing filenames with spaces in them, especially not when you're iterating over a file to grab the names. Unfortunately /bin/csh is mostly optimized for interactive use and not so much scripting. Resulting in a somewhat "messy" script.

It works, no problem, but produces some junk output and that will be noticeable when used with cron. Problem being that fuser sends output to both /dev/stdout _and_ /dev/stderr. And unfortunately csh can only redirect stdout _or_ stdout + stderr.

Still, this works.


```
#!/bin/csh

# Queue file (use full path!)
set queue=files.txt
# FTP host
set host=breve
# FTP username
set user=anonymous
# FTP password
set pass=password
# FTP upload path
set dir=/upload

## Don't change stuff beyond this line

set count=0
set size=`wc -l $queue | cut -d ' ' -f8`

if ($size == 0) then
        echo "No files to upload."                              > /dev/stderr
        exit 1
endif


if (`fuser $queue | cut -d ':' -f2`) then
        echo "Queue file currently in use, exiting!"            > /dev/stderr
        exit 1
endif

foreach a ("`cat $queue`")
        set check=$count;
        set input=`echo $a | sed -e 's/\ /\\\ /g'`
        ftp -Vu ftp://${user}:${pass}@${host}${dir}/"`basename $input`" "$input"
&& @ count++
        if (${check} == ${count}) then
                echo "Something went wrong with the upload."    > /dev/stderr
                break
        endif
end

@ newsize = ${size} - ${count}
tail -n $newsize $queue > `dirname $queue`/`basename $queue`_new
mv `dirname $queue`/`basename $queue`_new $queue
```
So... this script checks that the queue file isn't being processed, iterates over all the filenames in the queue file and optionally escapes spaces in the name ("my song" vs. "my\ song"), then uploads the whole kaboodle.

The rest is pretty much unchanged.

Note: don't try to start this with `sh ./<script>`. If you insist on doing it like that then use `csh`, but best is to set the execution bit and then start the script directly.

About that $a: That's the work of for (last script) and foreach. Iteration, goes over the output of a command one line at a time (that's the idea at least) and then iterates; constantly replacing the value of a for the next line in the output.

Ever ran into a situation where `rm *` would tell you "_too many arguments_"? Probably not but that can happen in large directories. Before I knew about xargs(1) I usually solved that problem like this: `for a in `ls`; do rm $a; done`. Removes all the files in the current directory, only this time one by one.


----------



## Sokonomi (Jul 29, 2018)

That script doesn't seem to work?
It belts through the queue regardless of ftp failure, removing all but the last line from files.txt (that indexing issue again).
For some reason that AND operator isn't working?

From what I can gather, theres only 2 options if I want a decent feedback on FTP;
using `grep` to fish out clues of what is happening with `ftp`,
or simply using a better FTP package that actually gives usable returncodes.


----------



## ShelLuser (Jul 30, 2018)

I kept output minimal, you can remove the -V parameter behind ftp which will re-enable full logging again.

Unfortunately I cannot reproduce any weird issues on my end; I actually do test this stuff before sharing 

There is one caveat which I discovered after posting: if there's a ' in the filename then the script will complain because it picks that character up literally.


----------



## Sokonomi (Jul 30, 2018)

So if it fails to connect, it wont bilk through the entire queue without stopping?
(Cause it does that for me)


----------



## ShelLuser (Jul 30, 2018)

Then this happens on my end:

```
$ ./upload
files.txt:
ftp: Can't lookup `account.no.way:ftp': hostname nor servname provided, or not known
ftp: Can't connect or login to host `account.no.way:ftp'
Something went wrong with the upload.
```
The script is very direct with this: the moment FTP exits with an error status of 1 or higher then count won't be increased in value and an error will be detected. There's really no way around that. I even tested it with using sh to start the script: `sh ./upload`, but that results in a simple hanging process which doesn't do anything.

So the best advice I can give you is to make sure the script file format is correct, csh is used or you use `chmod +x <script>` after which you start the script directly (`./script`).

Any problems beyond that I can only account to FreeNAS. That is the unknown factor here, at least for me.


----------



## Sokonomi (Jul 30, 2018)

I think I forgot to chmod the script, it appears to function now. Oops. 

Though I wonder if it would be better to see if theres a way for the script to adjust for changes to the queue file rather than just seeing if its in use. Because consider what happens if sonar shoves in a few new lines while the FTP script is the middle of uploading. The queue file will be longer than before the FTP script started, so it will end up cutting off more than just the ones it succeeded in uploading.  Would it fix that issue if I added `set size=`wc -l $queue | cut -d ' ' -f8`` after the loop part so it refreshes the size for determining how much of a tail it needs to cut? Or is it possible to give `head` a negative number, so it ll always trim the correct amount from the top regardless?


----------



## SirDice (Jul 30, 2018)

Sokonomi said:


> Or is it possible to give  head a negative number, so it ll always trim the correct amount from the top regardless?


head(1)


----------



## phoenix (Jul 30, 2018)

Now that you've done it the hard way and learned a tonne about scripting, install Plex on the NAS and use the auto-sync feature of the Plex client on the tablet.


----------



## Sokonomi (Jul 30, 2018)

phoenix said:


> Now that you've done it the hard way and learned a tonne about scripting, install Plex on the NAS and use the auto-sync feature of the Plex client on the tablet.


That would require one of those ridiculously expensive plex pass accounts, I think? Plex wont let me download the shows onto my tablet unless I do. Theres a way of having plex send new episodes down the tube automatically, serverside? I also don't see how it would be syncing 8Tb's worth of shows to a tablet that has just about 90Gb to play with. 

I might also need to circle back to the script not stopping even when I use chmod on it.




It will still churn through the list when it gets a server timeout, and act as if all of them were successful, deleting them off the queue file (except the last one, theres still an indexing issue).


----------



## ShelLuser (Jul 31, 2018)

Sokonomi said:


> I might also need to circle back to the script not stopping even when I use chmod on it.


Aha, now that makes somewhat more sense. I assume you defined those ports by using them behind your host specification? Although I don't really get why you'd host an FTP server on another port than 21, it seems plausible that the script picks up the colon and port in a different way (though I doubt it). Anyway, this is a situation I can easily test with on my jail.


----------



## Sokonomi (Jul 31, 2018)

It works when the server is actually up.  I was just testing what it does with errors.
I haven't tried what it does when the server actually kicks back a "disk full" error, but ftp(1) somehow gives an exit status 0 when the connection fails.
Ive googled it a bunch of ways, and I keep coming up with people using `grep` to get FTP status.
But by all accounts, the exit code should not be 0 when it comes out of a failed connection, right?

*EDIT:*
Ive kept on digging and found that shell will still think ftp(1) ran successfully regardless of it connecting or not (it did its job after all).
I found an alternative solution called NcFTP which might help me get a better handle on this thing.

An example would be;

```
ncftpput -u $user -p $pass -P $port $host $dir $a
```
It returns a usable diagnostic code:
 0 Success.
 1 Could not connect to remote host.
 3 Could not connect to remote host - timed out.
 4 Transfer failed.
 5 Transfer failed - timed out.
 6 Directory change failed.
 7 Directory change failed - timed out.
 8 Malformed URL.
 9 Usage error.
10 Error in login configuration file.
11 Library initialization failed.
12 Session initialization failed.

Basically anything but 0 still means it screwed up, but perhaps this is more predictable?
Code 4 might be used to delete last attempted file on the remote side as well,
or else it would leave incomplete files on the server, I think.


----------



## Chris_H (Jul 31, 2018)

Why don't you just awk(1) the STATUS returned from the ftp(1) command/run for each file?
Seems like it's be real easy. Say; use a conditional `while` / `if` / `unless`
In Perl speak:

```
unless ( $STATUS ) { do-something; } else { $VOMIT; }
```
where `$STATUS` was defined as the return code (number) for an ftp fail.
You could `awk` the status number returned from the ftp command(s), and place the failure number in `$STATUS`. But dropping any other ftp status number.

Just a thought. 

--Chris


----------

