# rsync running when ssh pipe broken



## mefizto (Jun 11, 2018)

Greetings all,

I use rsync over ssh to back-up to a remote server.  In irregular intervals the backup progress stalls, and the _remote _side reports: 
	
	



```
packet_write_wait: Connection to XXX.XXX.XXX.XXX port XXXX: Broken pipe
```
  However, when I check, judging by `top` and the exit status code, the rsync still runs.  Consequently, my script that is watching for the rsync exit status code to report success or to restart rsync never executes.

I do not have root privileges at the remote side; however, I can ask to run specific commands if they could help to analyse the network problem.  No irregularity appear on the client side.

Alternatively, is there a way to (1) detect that the ssh pipe is broken, so that rsync can be  can killed and restarted, or (2) have the ssh pipe breaking automatically kill rsync so that this event could be detected and rsync restarted by the script?

Kindest regards,

M


----------



## phoenix (Jun 11, 2018)

How are you running rsync over ssh? Via the *--rsh* option or via a pipe into ssh or something else?


----------



## mefizto (Jun 11, 2018)

Hi phoenix,

`rsync -avz -e ssh /path/to/source /path/to/target`.

Are there other ways?

Kindest regards,

M


----------



## phoenix (Jun 11, 2018)

No, that's the normal way. Just wanted to make sure.

How bizarre. I've yet to run into a situation like yours, when the ssh connection dies but rsync doesn't exit.


----------



## aragats (Jun 11, 2018)

I have had problems with *ssh* when one side had lower MTU, and _*ssh*_ packets got fragmented and it eventually failed: open sessions looked frozen.
Of course, I'm not sure whether that involved in your case, but you can try setting lower MTU manually, e.g. `ifconfig ue0 mtu 1400`.


----------



## mefizto (Jun 12, 2018)

Hi @phoenix,

yes, I though that since the rsync' transport depends on ssh, once the ssh pipe breaks, the rsync detests it and exits.  But no.  And it happened several times, so it is not a random occurrence.

Hi aragats,

I will try your suggestion, if my other idea I am testing will not have solved the problem.

Kindest regards,

M


----------



## bangmyhead (Jun 15, 2018)

mefizto said:


> Greetings all,
> 
> I use rsync over ssh to back-up to a remote server.  In irregular intervals the backup progress stalls, and the _remote _side reports:
> 
> ...



I do not know if it works for you but in my case I have mounted all my servers shares in a folder and then I use normal `rsync` from a folder to another.


----------



## mefizto (Jun 17, 2018)

Greetings all,

I have not solved the asked for problem, but solved the ssh pipe breakage by periodically sending a keep-alive packet to the server.

Kindest regards,

M


----------



## Lamia (Jun 17, 2018)

mefizto said:


> but solved the ssh pipe breakage by periodically sending a keep-alive packet to the server.


Autossh might prove useful here!


----------



## mefizto (Jun 20, 2018)

Hi Lamia,


Lamia said:


> Autossh might prove useful here!


I am aware that my solution does not address the underlying issue.  However, it seems that your proposed solution does not either.  I fail to see any advantage of installing additional software over my solution, which uses a configuration option of the rsync.

Could you please elaborate?

Kindest regards,

M


----------



## SirDice (Jun 20, 2018)

mefizto said:


> I have not solved the asked for problem, but solved the ssh pipe breakage by periodically sending a keep-alive packet to the server.


I suspect there's a firewall state timing out that's causing the connection to be dropped. If the firewall drops (instead of block) subsequent packets it can take a while for SSH to break off the connection as it will retry sending the packets a couple of times.  The 'trick' is to send regular keep-alive messages which will prevent the firewall state from timing out. Setting ServerAliveInterval and ServerAliveCountMax can help detect those dropped sessions sooner.


----------



## mefizto (Jun 20, 2018)

Hi SirDice,

that is exactly what I have concluded and done, and have not had the ssh to break sice.   Hence my question to Lamia.

Kindest regards,

M


----------

