# Pros/cons of SSH / netcat ZFS replication?



## littlesandra88 (Jan 16, 2013)

Hi All =)

What are the pros and cons of ZFS replication over SSH and netcat?

I see about 5x speed improvement when using netcat, but are the other things I should be awere of? E.g. are one better then the other when it comes to fault tolerance?

Hugs,
Sandra =)


----------



## SirDice (Jan 16, 2013)

littlesandra88 said:
			
		

> I see about 5x speed improvement when using netcat, but are the other things I should be awere of?


It's unencrypted, which is why SSH is slower. There's also no authentication.


----------



## cpm@ (Jan 16, 2013)

My suggestion is a cool resource, if you are looking a secure transport provider for logging. Try net/cryptcat the standard netcat enhanced with twofish encryption.


----------



## phoenix (Jan 16, 2013)

If running FreeBSD 8.x, install security/openssh-portable and enable HPN.

If running FreeBSD 9.x, add the following to /etc/make.conf and rebuild the world (or at least the OpenSSH parts):

```
### HACK until PR 163095 is committed
### http://www.freebsd.org/cgi/query-pr.cgi?pr=163095
.if ${.CURDIR:M/usr/src/secure/*}
CFLAGS+=-DNONE_CIPHER_ENABLED
.endif
```

Then you can use the None cipher in SSH, which gets you the best of all worlds:

encrypted connection
encrypted authentication
unencrypted datastream
bigger buffers, and auto-tuning buffers, for SSH connection

Note:  don't use an unencrypted transport medium across the Internet, or a network you don't control/trust.

Using HPN-enabled OpenSSH and the None cipher, we're able to saturate (980 Mbps) a gigabit link between two ZFS hosts for a ZFS send/recv operation.  With HPN and normal encryption, we get around 400-ish Mbps.  And with non-HPN (aka "normal) and normal encryption, we had trouble getting to 300 Mbps.  Between the same two servers.


----------



## throAU (Jan 17, 2013)

You can turn on compression with ssh (-C), which may help replication speed/bandwidth consumption over ssh, depending on your network bandwidth vs. CPU processing power.

As above, I wouldn't do unencrypted traffic between two machines for replication unless the network was local and/or otherwise secured such as via IPsec.


----------



## littlesandra88 (Jan 17, 2013)

Hi All,

Thanks a lot for the many details solutions! I will try them out now.

It is on a 10Gbit closed network, so it should be interesting to benchmark =)


----------



## Sebulon (Jan 17, 2013)

phoenix said:
			
		

> ```
> ### HACK until PR 163095 is committed
> ### http://www.freebsd.org/cgi/query-pr.cgi?pr=163095
> .if ${.CURDIR:M/usr/src/secure/*}
> ...



I had no clue about this. I am so going to try this out on our machines to test the difference Thanks man!

/Sebulon


----------



## phoenix (Jan 17, 2013)

throAU said:
			
		

> You can turn on compression with ssh (-C), which may help replication speed/bandwidth consumption over ssh, depending on your network bandwidth vs. CPU processing power.



This really depends on the speed difference between your disks and your network.  On our ZFS storage pools (4x 6-disk raidz2), SSH compression dropped throughput **a lot**.  The pool is "multi-threaded" (striped across multiple vdevs) but the SSH compression is single-threaded.  Even with a 2.0 GHz Opteron 6100-series CPU (8x magny-cours cores).  Disabling SSH compression sped things up.


----------



## cbunn (Jan 17, 2013)

But if one is transferring between two systems on a home network, would forgoing both authentication and encryption by using netcat be a bad idea? I'd rather not slip a hack into my SSH server if I don't have to.


----------



## throAU (Jan 23, 2013)

phoenix said:
			
		

> This really depends on the speed difference between your disks and your network.  On our ZFS storage pools (4x 6-disk raidz2), SSH compression dropped throughput **a lot**.  The pool is "multi-threaded" (striped across multiple vdevs) but the SSH compression is single-threaded.  Even with a 2.0 GHz Opteron 6100-series CPU (8x magny-cours cores).  Disabling SSH compression sped things up.



Yeah, as I said, depends on bandwidth vs CPU.

If you're doing replication over, say, a 512k frame-link (or over the internet) like I often have to deal with, compression is a good thing   Also, if speed is not important but bandwidth cost is (like 3G networks, as I also have to deal with ), compression helps.


On 10GB-e LAN, not so much.


----------



## Sebulon (Jan 24, 2013)

Hey all,

I just wanted to report that after recompiling two of our storage systems, sender and reciever, reconfigured sshd_config to allow the None cipher etc. and using the appropriate swtiches along with the ssh command, our zfs replications are now pushing 1GbE to the limit

`# zfs send pool/fs@snap | ssh -oNoneEnabled=yes -oNoneSwitch=yes -oHPNBufferSize=8192 username@host zfs recv pool/bakfs`

@phoenix

Is the HPN-switch really neccessary, since the buffer size is already specified in each sshd_config? And is there any way to write down the switches as default values in systemÂ´s, or userÂ´s ssh_config so that you donÂ´t have to specify the switches in the commands all of the time? I havenÂ´t been able to find any examples of it "out there"...

/Sebulon


----------



## phoenix (Jan 24, 2013)

The buffer isn't needed, as it's listed in the config file.

The None options must be specified on the client command-line each time, to make sure you really want it. A shell alias could work around that, though.


----------



## Sebulon (Jan 24, 2013)

> to make sure you really want it



Yeah, thatÂ´s what I thought as well. Thanks for the clarification! And thanks again for the hint, itÂ´s working wonders for our backuping

/Sebulon


----------

