# Recursive ftp / http backup with wget | tar | bzip2



## Leander (Apr 30, 2014)

Hi,

I want to recursively back up a directory lying on a FTP server. Also I want to compress it right away. All this sounds quite easy  but I want to optimize it a little by using pipes in order to save steps in between downloading and compressing. I was hoping for something like this:


```
wget -O - -q --no-host-directories --no-check-certificate --no-parent --recursive --user="${SRC_USR}" --password="${SRC_PWD}" "${SRC_URL}" | tar -c - | bzip2 -9 - > "${DST_DIR}/${DATE}.tbz2"
```

Unfortunately I can't get it working - it throws

```
tar: no files or directories specified
```
;/ Any ideas on how to accomplish this?

Thanks


----------



## Leander (Apr 30, 2014)

No ideas?


----------



## wblock@ (May 1, 2014)

`tar` needs a -f option.


----------



## Leander (May 1, 2014)

And therewith impossible for what I want, or am I misunderstanding you?


----------



## wblock@ (May 1, 2014)

```
tar -c -
```
needs to be

```
tar -c -f -
```


----------



## Leander (May 1, 2014)

Thanks for your reply. Unfortunately it doesn't work either.


```
wget -O - -q --no-host-directories --no-check-certificate --no-parent --recursive --user="${SRC_USR}" --password="${SRC_PWD}" "${SRC_URL}" | tar -c -f - | bzip2 -9 - > "${DST_DIR}/${DATE}.tbz2"
tar: no files or directories specified
```

Any other ideas?


----------



## kpa (May 1, 2014)

The problem is that tar(1) wants a list of files/directories on the command line as the source for the to be created archive, it can not take its input data from stdin when creating archives.


----------



## Leander (May 1, 2014)

Alright ;/
Could this maybe be solved with something else like e.g: 7z(1), 7za(1), cpio(1) or pax(1) instead of traditional tar(1)?
... would just be nice to save this additional steps in between 
Thanks


----------



## usdmatt (May 1, 2014)

I may be missing something as I don't work with wget much, but as I see it, the main problem you have is that the wget command is just going to output the content of the files, one-by-one. (If I'm wrong then the rest of my reply is probably irrelevant but the only conclusion I can come to is that wget is outputting the files to STDOUT)

You can pass that data directly into a compression program, but if you restore it, you'll just get one big file containing all the files that were backed up, one after the other, which would be a nightmare (if not impossible) to split up again.

Tar can take input from STDIN, but it expects to be passed a list of filenames, not actual data, as its primary function is to concatenate files whilst keeping their path and other metadata intact so the original structure can be recreated. It can't get this information if you just pass it the raw data.

You may have to rethink your single command plan (especially the way wget is used) as I can't see any way to get an archive which will allow you to store and recreate the original folder structure if you're just pulling a stream of the file contents.


----------



## Leander (May 1, 2014)

Yes, your assumption sounds quite like waht I expirienced. Passing the output directly to a compression app would work - but de-compressing would result in one big and unusable file, since the mapping of file and folder boundaries goes lost.
So I guess I can use && instead of pipes - haha


----------

