# Transferring Large Files Over Network



## Ruler2112 (Nov 9, 2011)

My old slackware 10 box is dying, so I built a FreeBSD box to replace it.  I have a great amount of data on the slackware box, so I got it running (no easy feat) and after setting up samba on the BSD machine, mounted the shared drive and started a cp -R on the directory last night and let it run overnight.  Unfortunately, it bombed out in short order because it hit upon a file larger than it could handle.  (I believe I have one about 80 gig in there...)  If I'm not mistaken, samba emulates a FAT file system and is subject to those limitations even though the underlying file system is ZFS.

I've googled and found rcp, scp, and rsync as common methods of transferring files over the network.  Never having used any of them, I don't know if the same or a similar limitation will be present in them.  Further, none of them are set up on either my slackware or BSD boxes.  I also thought of tar-ing up the whole directory and splitting it into 2-gig chunks, but honestly don't think this would be a good idea - not only would be it exceedingly slow, but I'd have to monitor it closely because there's not enough drive space to do it all at once.  This is hopefully going to be a one-time thing, so I don't want to spend a ton of time setting up/configuring a complex system to do it if avoidable.  I'm in a secure local environment and can safely drop the firewalls on both machines if needed.

Is anybody familiar with the rcp/scp/rsync utilities?  Are they capable of transferring very large files over the network?  Are there any other tools/services I should consider using instead?


----------



## Ruler2112 (Nov 9, 2011)

Just discovered that samba also doesn't allow non-FAT characters in file names, so I can't even use it to transfer the directories containing only small files...


----------



## wblock@ (Nov 9, 2011)

Both rsync and scp are capable of transferring large files reliably.  scp is already present on FreeBSD, and I'd be surprised if it's not on slackware.  There's no setup needed, it just uses sshd.  So does rsync, which is an easy install and smarter than scp.


----------



## Ruler2112 (Nov 9, 2011)

Thanks for the quick response wblock.

I don't have ssh configured on either end.  I've tried setting it up a couple times in the past on different machines, but never did get it working and couldn't seem to figure out what I was doing wrong.  Since I'm in a secure environment, I just uncommented the telnetd lines in the inetd config file and was done with it...

However, I did find a solution to the problem.  Wouldn't you know it - I googled for close to an hour before posting and 5 minutes afterward, found this.  At the bottom, I found a set of commands that's dumping data through the network as I write this.  It hasn't hit any of the excessively large files yet, but since it's not using any non-*nix tools, I don't believe there should be a problem.


```
bsd# nc -l 111.222.333.444 1234 | tar -xf -

slackware# tar -cvf - /dir | nc 111.222.333.444 1234
```

The first command, run on the BSD box, starts netcat in listening mode.  It pipes what it gets to tar, which uncompresses it to the directory you're in.

The second command, run on the slackware machine, uses tar to compress the directory to standard out and pipes it into netcat, which connects to the BSD box and dumps it over the network.

So far, it's running MUCH faster than samba.


----------



## wblock@ (Nov 9, 2011)

/etc/rc.conf

```
sshd_enable="YES"
```

is all that's needed on FreeBSD.  telnetd is not needed.  Then, from the slackware computer:
`% rsync -avz /dir/ [email]freebsduser@targetcomputer[/email]:/dir/`


----------



## SirDice (Nov 10, 2011)

The netcat trick is nice but not really secure. You can use the same kind of technique using ssh:

`# ssh somebox tar -C /some -cf - dir | tar -C /other -xvf -`

This will _pull_ /some/dir/ from the host somebox to the local /other/dir/.

`#  tar -C /some -cf - dir | ssh somebox tar -C /other -xvf -`

This will _push_ the local /some/dir/ to the remote /other/dir/ on host somebox.

But since you are on a secure network why not simply use NFS?


----------



## Martillo1 (Dec 2, 2011)

Samba also emulates other filesystems apart from FAT. I set up a samba server which presented a NTFS share so that my wife could store in my FreeBSD system her stuff from her Windows 7 machine.

Yeah, love make us do the strangest things :e


----------



## phoenix (Dec 2, 2011)

Samba doesn't "emulate" a filesystem.  It doesn't "emulate" FAT or NTFS or anything like that.  It *is* a network filesystem using SMB to share files.


----------

