# Unable to ssh (locked out) after upgrade from 10.1 to 10.3



## molofishy (Sep 19, 2016)

I followed the update instructions here to update from 10.1 to 10.3. My only way of accessing this server is via ssh, and I have now been locked out (the ssh address is no longer known by my PC). Are there any steps to take / configurations to check to prevent this from happening again? I cannot afford to be locked out every time I update the version — I will need to take the server to my work and connect it to a screen to be able to fix this. 

Thanks,
Oliver


----------



## Remington (Sep 19, 2016)

molofishy said:


> I followed the update instructions here to update from 10.1 to 10.3. My only way of accessing this server is via ssh, and I have now been locked out (the ssh address is no longer known by my PC). Are there any steps to take / configurations to check to prevent this from happening again? I cannot afford to be locked out every time I update the version — I will need to take the server to my work and connect it to a screen to be able to fix this.
> 
> Thanks,
> Oliver



`Mergemaster` probably overwrote the file /etc/ssh/sshd_config when you upgraded FreeBSD. You need to verify it before restarting the server after the upgrade.


----------



## molofishy (Sep 20, 2016)

Remington said:


> `Mergemaster` probably overwrote the file /etc/ssh/sshd_config when you upgraded FreeBSD. You need to verify it before restarting the server after the upgrade.



This sounds to me like a major issue with updating. Why should the `sshd_config` file be overwritten? I strongly suggest something about this be added to the Upgrading FreeBSD page, unless I am missing something?

Thanks,
Oliver


----------



## Remington (Sep 20, 2016)

molofishy said:


> This sounds to me like a major issue with updating. Why should the `sshd_config` file be overwritten? I strongly suggest something about this be added to the Upgrading FreeBSD page, unless I am missing something?



When you run `mergemaster -PFUi`, it will ask you if you want to install or delete temporary file, or merge both temporary and installed files.  If you want to keep your modified config file then you would select to delete the temporary file as it is default config and you don't want to replace your modified config with temporary config.  The merge option will display left and right columns and it will ask you which you want to keep by pressing 'l' which is left or 'r' which is right.

I think that's where you got confused about what is temporary and installed file so you accidentally selected to install temporary which overwrote your sshd_config with the default config.  That would explain why you got locked out.

Mergemaster is important upgrade tool and you need to understand it well otherwise there will be unintended consequences.

Please read the man doc on `mergemaster`.


----------



## SirDice (Sep 20, 2016)

I wonder why you can't access it though, by default sshd(8) listens on all addresses. What else was changed in sshd_config?


----------



## kpa (Sep 20, 2016)

I think the handbook could have a short example on merging with mergemaster(8). The section "23.6.4. Merging Configuration Files" does give an idea how it works but doesn't show what the procedure actually looks like.


----------



## ANOKNUSA (Sep 20, 2016)

molofishy said:


> I strongly suggest something about this be added to the Upgrading FreeBSD page, unless I am missing something?



While the instructions in the _Handbook_ could be a little clearer, you admit yourself that you didn't actually follow the instructions in the _Handbook_. Hurling blame at the people who wrote the instructions you didn't use is, in polite terms, "poor form."


----------



## Remington (Sep 20, 2016)

kpa said:


> I think the handbook could have a short example on merging with mergemaster(8). The section "23.6.4. Merging Configuration Files" does give an idea how it works but doesn't show what the procedure actually looks like.



I agree.  A screenshot will help.

It think FreeBSD Handbook should allow moderator-approved FreeBSD users to edit the documents similar to Wikipedia.  That will make editing the docs much easier and more current.


----------



## SirDice (Sep 20, 2016)

Updates to the handbook are very much welcomed:

FreeBSD Documentation Project Primer for New Contributors

And a wiki isn't going to work. The handbook (and other documentation) is done in a way that allows exports to various other formats, like the HTML on the site, PDF and numerous other document formats.


----------



## Remington (Sep 20, 2016)

SirDice said:


> Updates to the handbook are very much welcomed:
> 
> FreeBSD Documentation Project Primer for New Contributors



I understand that but not everyone likes to checkout using CVS.  What I meant is online editing like Wikipedia with revision control by only approved FreeBSD users.  There has to be a way to simplify the editing process.

There could be two documentations: Official FreeBSD Handbook (CVS) and FreeBSD Users Handbook (Wikipedia style).


----------



## SirDice (Sep 20, 2016)

Remington said:


> There could be two documentations: Official FreeBSD Handbook (CVS) and FreeBSD Users Handbook (Wikipedia style).


Sure, who's going to keep them the same?


----------



## Remington (Sep 20, 2016)

SirDice said:


> Sure, who's going to keep them the same?



It's not going to be the same but wikipedia style documentations will contain more information and how-to which is severe lacking in official Handbook. Which do you think will likely receive more contributors to update the documentations?  CVS or Wikipedia style Handbook?  Also, the official Handbook are in several languages and they're not the same.

Personally I think CVS is great for code maintainers but its terrible for Handbook documentations.  Programmers and editors are two different things.  Editors won't know how to use CVS as its designed for programmers and they won't waste time trying to figure out how to install and use CVS.  That's why I said wikipedia style or online documentation editor are way much easier and will receive more contributors to update the docs.

PHP have their wiki page (wiki.php.net) and Docbook Online Editor (edit.php.net).

May I suggest adding online editing software and let FreeBSD users fill in the rest.  After-all, its the contributors who contributes to FreeBSD community, therefore, need to make the process easier for FreeBSD contributors.

CVS Handbook documentation is becoming an old-school (no offense intended).  That's why molofishy was confused because of lack of clarifications on using `mergemaster`.  That's something FreeBSD committee will have to think about.  We're not in 20th century anymore.


----------



## SirDice (Sep 20, 2016)

Lets not take this thread further offtopic than it already is. If you have suggestions regarding the documentation process please open a new thread, I'm sure people like wblock@ (regular documentation submitter) will chime in.


----------



## molofishy (Sep 21, 2016)

Finally got the server to a screen. Here are the contents of /etc/ssh/sshd_config

Again, anything odd here was simply a consequence of updating from 10.1 to 10.3 following the standard update instructions. /etc/rc.conf was unchanged. I do not know what /etc/ssh/sshd_config looked like before so I do not know what it should look like (will need to do some reading). I did not use mergemaster and maybe I should have? 

I am obviously not interested in blaming anyone (no idea why that was raised in this thread). I just want to get to the bottom of this to help myself and other users, and am happy to update the handbook once a solution is found. 

Thanks,
Oliver


----------



## kpa (Sep 21, 2016)

Using mergemaster(8) is NOT optional if you use the source based update/upgrade. You have to run it every single time you have installed a new world.


----------



## molofishy (Sep 21, 2016)

I used the "FreeBSD Update" option. I did the following in the order shown:

`freebsd-update fetch
freebsd-update install
freebsd-update upgrade -r 10.3-RELEASE
freebsd-update install
shutdown -r now
freebsd-update install`


----------



## Remington (Sep 21, 2016)

You're welcome to post your sshd_config

If building from source, do this:

```
cd /usr/src
make buildworld
make buildkernel KERNCONF=DBSDV
make installkernel KERNCONF=DBSDV
<reboot>
mergemaster -p
cd /usr/src
make installworld
mergemaster -PFUi
yes | make delete-old
yes | make delete-old-libs
<reboot>
```

Anyway, you actually installed FreeBSD 10.3 twice. 


```
freebsd-update fetch
freebsd-update install
freebsd-update upgrade -r 10.3-RELEASE
freebsd-update install
shutdown -r now
freebsd-update install
```

First two lines are for upgrading from 9.3, 10.0, 10.1, and 10.2 to 10.3.  This is better for minor upgrades.
Third line is for upgrading from 9.2 and older to 10.3-RELEASE. This is better for major upgrades.

So, either one will work in your case.

Did you run `pkg upgrade` to update all the installed packages after you did the major upgrade?  This is required after major upgrades.


----------



## kpa (Sep 21, 2016)

molofishy said:


> I used the "FreeBSD Update" option. I did the following in the order shown:
> 
> `freebsd-update fetch
> freebsd-update install
> ...



You're not using the source update/upgrade but the freebsd-update(8) method. Don't mix the two. When using freebsd-update(8) the update tool is supposed to handle the merging of the configuration files and present you with the merge options if there is a need for merging. You don't use mergemaster(8) when using freebsd-update(8).


----------



## fulano (Sep 21, 2016)

Remington said:


> make buildkernel KERNCONF=DBSDV make installkernel KERNCONF=DBSDV



KERNCONF is optional.


----------



## kpa (Sep 21, 2016)

fulano said:


> KERNCONF is optional.



Not quite if you write it out as intended:

`make buildkernel KERNCONF=DBSDV`

`make installkernel KERNCONF=DBSDV`

Without repeating it on the second invocation the installkernel target would try to install the GENERIC kernel.

However this is enough to achieve the same:

`make buildkernel installkernel KERNCONF=DBSDV`


----------



## Remington (Sep 21, 2016)

kpa said:


> `make buildkernel installkernel KERNCONF=DBSDV`



What about this?

`make -j8 buildkernel installkernel KERNCONF=DBSDV`


----------



## molofishy (Sep 21, 2016)

Remington: oh I installed it twice? whoops! . btw, I provided a link to photographs of my sshd_config file in my previous post. I seems the entire thing was overwritten. I set 
	
	



```
PasswordAuthentication
```
 to yes and added my username to 
	
	



```
AllowUsers
```
. Now I can ssh again like before!

Although I'm still curious as to why the file was wiped to default. I must have done something unusual. Perhaps it was something to do with the double install... 

I did not run pkg upgrade after the install. I'll remember this too.

kpa: I did not see the options. I will keep an eye out next time.

If you need more information, I am happy to provide.


----------



## Remington (Sep 21, 2016)

It's more likely you misunderstood the merge process when you upgraded FreeBSD.  There's no way the upgrade will secretly overwrite config files without your permission so you must have missed or misunderstood when it prompted you about sshd_config.


----------



## kpa (Sep 21, 2016)

Remington said:


> What about this?
> 
> `make -j8 buildkernel installkernel KERNCONF=DBSDV`



It might work but then again the installkernel target probably does not like multiple threads and of course does not benefit from them because the installation has to be done in very strict order.


----------



## SirDice (Sep 21, 2016)

molofishy said:


> I set PasswordAuthentication to yes and added my username to AllowUsers. Now I can ssh again like before!


The default sshd_config allows anyone with a valid user account to login using passwords. I don't understand why those changes were necessary to allow you to login.


----------



## Remington (Sep 21, 2016)

I don't see how those changes will make a difference unless if trying to login as root?

It's hard to guess without seeing your whole sshd_config file before you made the changes.


----------



## molofishy (Sep 21, 2016)

SirDice said:


> The default sshd_config allows anyone with a valid user account to login using passwords. I don't understand why those changes were necessary to allow you to login.



SirDice Hmm. When I remove a user's name from the AllowUsers list, they no longer have access. I just confirmed this again. There are only 3 lines uncommented in my /etc/ssh/sshd_config, the two I mentioned above and

```
Subsystem sftp /usr/libexec/sftp-server
```
Perhaps a different file was modified during the upgrade, not /etc/ssh/sshd_config. Are you saying the default sshd_config that I had should have been sufficient? Fyi, I am using passwordless ssh-keys for now (don't worry, I will add a password to my ssh-key)  -- if such information is relevant for diagnostics.


----------



## Remington (Sep 21, 2016)

There is no need to add password to your ssh-key.  It's already far more secure than logins using a password.  I don't use passwords with my ssh-key since nobody else have access to my computer.  However, its up to you how you want to set it.


----------



## kpa (Sep 21, 2016)

Remington said:


> There is no need to add password to your ssh-key.  It's already far more secure than logins using a password.  I don't use passwords with my ssh-key since nobody else have access to my computer.  However, its up to you how you want to set it.



No, please. On a standard system your secret keyfile is readable by any process run under your user account and you'll put that secret key under an enormous risk if your key file is not protected by a PIN code (well password but technically it functions as a PIN code). Any programming error in an application that allows a remote attacker to read arbitrary files from your home directory will be able to steal your secret key.

This is the very reason why web browsers have moved to sandboxed extensions. The extensions are run in a "fake" environment where only the necessary part of the environment are present and sensitive parts of the users home directory are not available.


----------



## Remington (Sep 21, 2016)

kpa said:


> No, please. On a standard system your secret keyfile is readable by any process run under your user account and you'll put that secret key under an enormous risk if your key file is not protected by a PIN code (well password but technically it functions as a PIN code). Any programming error in an application that allows a remote attacker to read arbitrary files from your home directory will be able to steal your secret key.
> 
> This is the very reason why web browsers have moved to sandboxed extensions. The extensions are run in a "fake" environment where only the necessary part of the environment are present and sensitive parts of the users home directory are not available.



I'm aware of this.  I'm using Mac and its far less likely will be hacked or infected as compared to Windows.  Also, if you set `AllowUsers you@192.168.0.0/16` in sshd_config and that will only allow SSH connections from that remote IP address.  Configuring firewall is another way of restricting SSH connections from specific IP range and I use non-standard port for SSH too.

I don't disagree that using password with ssh-key is good security but unnecessary if your computer is well protected.  I trust Mac more than Windows.


----------



## molofishy (Sep 22, 2016)

SirDice and Remington, you are correct that `AllowUsers` is not required. I was confused because when that command is used, it grants permission to only the users listed. Although when it is removed completely, all users have access (I thought no uses would have access). So now the only thing I have actually added to the entire file since being locked out is `PasswordAuthentication yes`. Or is this also somehow not required? If so then I'm a little confused as to why I was ever "locked out"...


----------



## SirDice (Sep 22, 2016)

molofishy said:


> So now the only thing I have actually added to the entire file since being locked out is  PasswordAuthentication yes. Or is this also somehow not required?


It shouldn't be needed actually. All my systems run the default sshd_config and I can login using any valid user. Both with 'regular' passwords and by using using keys.


----------



## molofishy (Sep 22, 2016)

SirDice said:


> It shouldn't be needed actually. All my systems run the default sshd_config and I can login using any valid user. Both with 'regular' passwords and by using using keys.



Interesting. Thanks a lot for you help. I think I'm too nervous to try it though. I don't want to get locked out again.  Do you know how I would disable regular passwords and only allow keys?


----------



## SirDice (Sep 22, 2016)

To allow only keys you need to set PasswordAuthentication and ChallengeResponseAuthentication to no. Which provides an answer to the first question too, PasswordAuthentication is set to yes by default, setting it explicitly in the config doesn't actually change it.


----------



## molofishy (Oct 1, 2016)

SirDice said:


> To allow only keys you need to set PasswordAuthentication and ChallengeResponseAuthentication to no. Which provides an answer to the first question too, PasswordAuthentication is set to yes by default, setting it explicitly in the config doesn't actually change it.



SirDice, cool, but I assume I need to make sure I have already scp'd my public sshkey to authorized_keys on the server? If not, I'm locked out after I make those two changed you suggest.


----------



## Remington (Oct 1, 2016)

You can do this:

1) Login using your regular password (never log off).
2) Edit sshd_config and set PasswordAuthentication and ChallengeResponseAuthentication to no and save changes.
3) Reload SSH daemon by doing this: `service sshd reload`
4) Make another login using your ssh key.

When sshd reloads it won't kick or kill your current connection.


----------



## leebrown66 (Oct 2, 2016)

If you are doing this remotely, the safest way is to fire up a _second_ sshd daemon manually with a separate config file using a different port and try to login via that.  If it fails, restart with lots of debug (-d) turned on and find out why.  Fix and repeat.
Once you have verified the configuration works, move the test-config to the production-config, then restart the sshd daemon.  That way you never have to lose your working sshd until you are confident you can switch


----------



## fullauto2012 (Mar 17, 2017)

Anyone figure out a way to fix this?
As a result of this same problem, I now have to drive 13 hours to finish the upgrade...


----------



## molofishy (Mar 17, 2017)

I did not find the cause. My latest hypothesis is that the machine did not reboot properly. Try rebooting a couple times or get someone else to.


----------

