# Cannot connect via ssh to remote host following upgrade to 13.1



## byrnejb (Jul 11, 2022)

Following the upgrade to freebsd-13.1 I encounter the following when trying to connect via ssh:

```
# sshx -vvv vhost02.windsor
OpenSSH_7.9p1, OpenSSL 1.1.1l-freebsd  24 Aug 2021
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 3: Applying options for *
debug1: /root/.ssh/config line 54: Applying options for vhost02.windsor
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "vhost02.windsor" port 22
debug2: ssh_connect_direct
debug1: Connecting to vhost02.windsor [216.185.71.142] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9961 ms remain after connect
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9 FreeBSD-20200214
ssh_exchange_identification: Connection closed by remote host
```

The sequence of events was:


```
[root@vhost02 ~ (master)]# freebsd-version
13.0-RELEASE-p11
[root@vhost02 ~ (master)]# freebsd-update fetch
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p11.

WARNING: FreeBSD 13.0-RELEASE-p11 is approaching its End-of-Life date.
It is strongly recommended that you upgrade to a newer
release within the next 2 weeks.

[root@vhost02 ~ (master)]# freebsd-update upgrade -r 13.1-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic kernel/generic-dbg world/base world/lib32

The following components of FreeBSD do not seem to be installed:
world/base-dbg world/lib32-dbg

Does this look reasonable (y/n)? y

Fetching metadata signature for 13.1-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Fetching files from 13.0-RELEASE for merging... done.
Preparing to download files... done.
Fetching 5170 patches...
. . .
/var/db/etcupdate/current/root/.profile
/var/db/etcupdate/log
/var/db/mergemaster.mtree
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

[root@vhost02 ~ (master)]# /usr/sbin/freebsd-update install
src component not installed, skipped
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

[root@vhost02 ~ (master)]# shutdown -r now
Shutdown NOW!
shutdown: [pid 17591]
*** FINAL System shutdown message from root@vhost02.windsor.harte-lyne.ca ***

System going down IMMEDIATELY                                                

System shutdown time has arrived
Connection to vhost02.windsor.harte-lyne.ca closed by remote host.
Connection to vhost02.windsor.harte-lyne.ca closed.

[root@vhost01 ~ (master)]# sshx vhost02.windsor.harte-lyne.ca

[root@vhost02 ~ (master)]# /usr/sbin/freebsd-update install
src component not installed, skipped
Installing updates...Scanning //usr/share/certs/blacklisted for certificates...
Scanning //usr/share/certs/trusted for certificates...
Scanning //usr/local/share/certs for certificates...
 done.

[root@vhost02 ~ (master)]# pkg upgrade
Updating FreeBSD repository catalogue...
Fetching packagesite.pkg: 100%    6 MiB
```

The most recent entries in /var/log/messages are:

```
Jul 11 10:53:58 <user.notice> vhost02 pkg[32456]: postfix upgraded: 3.7.0_1,1 -> 3.7.2,1
Jul 11 10:55:44 <user.notice> vhost02 pkg[35276]: lsof reinstalled: 4.95.0,8 -> 4.95.0,8
Jul 11 10:58:57 <kern.info> vhost02 kernel: pid 1140 (spiped), jid 1, uid 0: exited on signal 11 (core dumped)
Jul 11 11:02:24 <auth.crit> vhost02 sshd[35668]: fatal: recv_rexec_state: parse config: incomplete message
Jul 11 11:30:07 <auth.crit> vhost02 sshd[36828]: fatal: recv_rexec_state: parse config: incomplete message
Jul 11 11:30:14 <auth.crit> vhost02 sshd[36829]: fatal: recv_rexec_state: parse config: incomplete message
Jul 11 11:51:51 <auth.crit> vhost02 sshd[36994]: fatal: recv_rexec_state: parse config: incomplete message
```

Following the second`freebsd-update install` other systems are unable to connect via `ssh` and instead receive the error above.

Researching this I attempted to determine if ssh was linked to a delete library using `lsof`.  However, this utility does not work on FreeBSD using ZFS, which this system is.  I have checked the keys in /etc/ssh/ and these remain unchanged.  There are no `Match User/Match Group` directives active in /etc/ssh/sshd_config.

Attempting to connect to this host via ssh does not cause any more entries to be logged in /var/log/messages.
I am maintaining the existing ssh connection used to install 13.1 as this host is a considerable (hours) distance from my location.  From this connection I can `ssh` out to the hosts that cannot `ssh` in.

What is wrong and how do I fix this?


----------



## SirDice (Jul 11, 2022)

byrnejb said:


> how to I fix this?


Reboot the host.


----------



## Eric A. Borisch (Jul 11, 2022)

Or `service sshd restart`.

From the release notes:



> After upgrading, sshd (from OpenSSH 8.8p1) will not accept new connections until it is restarted. After installing the new userland, either reboot (as specified in the source update procedure), or execute service sshd restart.


----------



## byrnejb (Jul 11, 2022)

Further info.  I started as second sshd instance running on 2222 and was able to connect locally from the existing sesion.  However, there is a firewall preventing me from connected from out side


----------

