# SSH client always using port 10000



## ronaldlees (Aug 8, 2016)

I suddenly have a problem where the local `ssh` clients are always using local port 10000 when connecting to remote `sshd` servers (remote servers are all on port 22).  What might that be?


----------



## derekschrock (Aug 8, 2016)

Look for Port in /etc/ssh/config or ~/.ssh/config
Check to see if ssh is an alias or shell function `type ssh`


----------



## ronaldlees (Aug 9, 2016)

derekschrock said:


> Look for Port in /etc/ssh/config or ~/.ssh/config
> Check to see if ssh is an alias or shell function `type ssh`



I see no "Port" specified in the config files, and `type ssh` =/usr/sbin/ssh.

What's odd is that I have two different computers doing the exact same thing.


----------



## derekschrock (Aug 9, 2016)

Can you post `ssh -vv` output?


----------



## ronaldlees (Aug 9, 2016)

Out of the blue, it's random again (random local ports).  So, the "-vv" output probably won't show us what was wrong.  What is particularly strange is that this happened to two different machines (one Linux, one BSD) - and was consistent over the past day or so (a good number of connections too).


----------



## SirDice (Aug 9, 2016)

Phishfry said:


> My config file is here:
> /etc/ssh/sshd_config


That's the config for the sshd(8) daemon, the ssh(1) client uses /etc/ssh/ssh_config.


----------



## SirDice (Aug 9, 2016)

ronaldlees said:


> I see no "Port" specified in the config files,


Port is to specify a _destination_ port, not a source port. As far as I know ssh(1) simply uses whatever random port the OS gives it. What does `sysctl net.inet.ip.portrange` output?


----------



## ronaldlees (Aug 9, 2016)

SirDice said:


> Port is to specify a _destination_ port, not a source port. As far as I know ssh(1) simply uses whatever random port the OS gives it. What does `sysctl net.inet.ip.portrange` output?




```
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
net.inet.ip.portrange.randomized: 1
net.inet.ip.portrange.reservedlow: 0
net.inet.ip.portrange.reservedhigh: 1023
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 10000
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.lowfirst: 1023
```

SirDice OK, I see the 10000 now.  I would expect the port would have been randomized within the range.  It definitely was not randomized throughout the day yesterday.


----------



## SirDice (Aug 9, 2016)

Yes, that at least explains the port 10000. But you are right, I would have expected the ports to be more random too. If you make two ssh(1) connections in short succession do they both use source port 10000?


----------



## ronaldlees (Aug 9, 2016)

SirDice said:


> Yes, that at least explains the port 10000. But you are right, I would have expected the ports to be more random too. If you make two ssh(1) connections in short succession do they both use source port 10000?



Indeed - I saw that happen a couple times. At least I think I did.  You know how it is, when you're first looking at something like this, and then you've cleared it off, and later ask yourself "Did I really see that?"


----------



## kpa (Aug 9, 2016)

ronaldlees said:


> Indeed - I saw that happen a couple times. At least I think I did.  You know how it is, when you're first looking at something like this, and then you've cleared it off, and later ask yourself "Did I really see that?"



One possible explanation is that you already have many outgoing connections that have reserved many ports and the randomization algorithm doesn't find a suitable source port and it falls back to the first one available.


----------



## ronaldlees (Aug 10, 2016)

kpa said:


> One possible explanation is that you already have many outgoing connections that have reserved many ports and the randomization algorithm doesn't find a suitable source port and it falls back to the first one available.



I can't say that I had sufficient connections to imagine the supply could have been exhausted 

The paranoid in me thinks about what a static source port might facilitate (like a tunnel back to me via an existing outgoing connection).  But, I didn't see any evidence of anything like that in the configuration on my machine, so no reason to think so.

It hasn't happened yet today AFAIK.  It's a mystery.


----------



## fossette (Aug 10, 2016)

Perhaps a randomless random generator hitting a state which always returns 0 (the first entry in the TCP Port list)?  If so, this seems to go back to normal after rebooting, or cause an event that regenerates the random generator seed.  I've seen this kind of 'stuck' behavior while implement my own random number generator algorythm.


----------



## Murph (Aug 10, 2016)

ronaldlees said:


> The paranoid in me thinks about what a static source port might facilitate (like a tunnel back to me via an existing outgoing connection).  But, I didn't see any evidence of anything like that in the configuration on my machine, so no reason to think so.



With the exception of a man-in-the-middle scenario, predictable ports should not make a tunnel to a different endpoint possible, as while they might be able to spoof an incoming packet, the outgoing packets won't reach them.  Note that they need to successfully predict not just the port, but also the TCP sequence numbers (which should start from a random point).  They would also face the near impossible challenge of actually breaking the SSH crypto protections which should resist even someone with true man-in-the-middle access to your TCP stream (as long as you always verify remote keys).

Personally, I wouldn't lose a great deal of sleep over it, unless I saw some other odd behaviour that seemed to relate to it.  Just check that you are up to date with security fixes, and chalk it up to one of those occasional random oddities.


----------



## ronaldlees (Aug 10, 2016)

fossette said:


> Perhaps a randomless random generator hitting a state which always returns 0 (the first entry in the TCP Port list)?  If so, this seems to go back to normal after rebooting, or cause an event that regenerates the random generator seed.  I've seen this kind of 'stuck' behavior while implement my own random number generator algorythm.



Yikes!  I would hope the RNG is not that bad!  I did run `dieharder` (in ports as math/dieharder ) - and it gave me a decent evaluation on the random number generator (although the author indicates that it's a non-exhaustive evaluation).  Given the disclaimer, does anyone know of similar tools I could use to test the RNG, to get a second opinion?


----------



## kpa (Aug 10, 2016)

Write a program that creates a large number of sockets and writes down the source ports the OS assigned to the sockets with its randomizer.


----------



## SirDice (Aug 10, 2016)

Murph said:


> Note that they need to successfully predict not just the port, but also the TCP sequence numbers (which should start from a random point).


Tricky to do indeed. Just contemplating here, but suppose the OPs random generator is somehow borked and it's causing source ports to not be random wouldn't that same lack of randomness affect the sequence numbers too? If one supposed random number isn't random, the others won't be either?


----------



## ronaldlees (Aug 10, 2016)

kpa said:


> Write a program that creates a large number of sockets and writes down the source ports the OS assigned to the sockets with its randomizer.



Good idea.  Do we even know whether or not the system RNG is used with port number allocation?


----------



## ronaldlees (Aug 10, 2016)

SirDice said:


> Tricky to do indeed. Just contemplating here, but suppose the OPs random generator is somehow borked and it's causing source ports to not be random wouldn't that same lack of randomness affect the sequence numbers too? If one supposed random number isn't random, the others won't be either?



I find it hard to believe that the RNG would do this, so I'm leaning towards your suggestion that "the OPs random generator is somehow borked" - probably via exploit or something, and that I should do a drop-kick on my current instances, reinstall the OS, and start over.


```
$ sysctl kern.random
kern.random.sys.harvest.swi: 1
kern.random.sys.harvest.interrupt: 1
kern.random.sys.harvest.point_to_point: 1
kern.random.sys.harvest.ethernet: 1
kern.random.sys.seeded: 1
kern.random.yarrow.slowoverthresh: 2
kern.random.yarrow.slowthresh: 128
kern.random.yarrow.fastthresh: 96
kern.random.yarrow.bins: 10
kern.random.yarrow.gengateinterval: 10
kern.random.live_entropy_sources:
kern.random.active_adaptor: yarrow
kern.random.adaptors: yarrow,dummy
```


```
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
```
What's the implication of the last two lines (from `net.inet.ip.portrange`?)


----------



## Murph (Aug 10, 2016)

ronaldlees said:


> What's the implication of the last two lines (from `net.inet.ip.portrange`?)



On 10.3:

```
# sysctl -d net.inet.ip.portrange
net.inet.ip.portrange: IP Ports
net.inet.ip.portrange.randomtime: Minimum time to keep sequental port allocation before switching to a random one
net.inet.ip.portrange.randomcps: Maximum number of random port allocations before switching to a sequental one
net.inet.ip.portrange.randomized: Enable random port allocation
net.inet.ip.portrange.reservedlow: 
net.inet.ip.portrange.reservedhigh: 
net.inet.ip.portrange.hilast: 
net.inet.ip.portrange.hifirst: 
net.inet.ip.portrange.last: 
net.inet.ip.portrange.first: 
net.inet.ip.portrange.lowlast: 
net.inet.ip.portrange.lowfirst:
```

Those descriptions suggest to me that it flips back and forth between sequential and random, presumably for performance reasons (random is expensive, compared to sequential).  I would not rush to any conclusions about the need to burn your system to the ground, certainly not without much more compelling evidence of a compromise.  I don't know the precise details underneath that behaviour, but the above certainly could explain something like you are seeing, and sequential could quite possibly result in the same port number appearing repeatedly in this sequence {open, close, open, close, …}, and an incrementing port number in {open, open, open, …, close, close, close}.


----------



## Murph (Aug 10, 2016)

Ahh, here we are it is documented in TFM, ip(4):


> Ports are allocated at random within the specified port range in order to
> increase the difficulty of random spoofing attacks.  In scenarios such as
> benchmarking, this behavior may be undesirable.  In these cases,
> net.inet.ip.portrange.randomized can be used to toggle randomization off.
> ...


----------



## kpa (Aug 10, 2016)

I wonder why UDP doesn't prefer more randomized source port selection, it is much more vulnerable to forgery *) by an active attacker than TCP which has as noted the sequence numbers as extra protection.


*) For example DNS used to use fixed source port 53 also for any outgoing queries, this was hopelessly insecure given how the DNS protocol works.


----------



## kpa (Aug 10, 2016)

kpa said:


> I wonder why UDP doesn't prefer more randomized source port selection, it is much more vulnerable to forgery *) by an active attacker than TCP which has as noted the sequence numbers as extra protection.
> 
> 
> *) For example DNS used to use fixed source port 53 also for any outgoing queries, this was hopelessly insecure given how the DNS protocol works.



Of course the random UDP source port applies only to UDP based protocols that have been designed in such a way or have been fixed to work that way. It's common especially on game protocols that use UDP as transport to just use a single UDP port for both directions, ingoing and outgoing. This is because it's much easier to program a game server with just a single UDP listening port. For the same reason we still have the NTP protocol that uses only one UDP listening port 123 regardless of direction.


----------



## Murph (Aug 10, 2016)

kpa said:


> For the same reason we still have the NTP protocol that uses only one UDP listening port 123 regardless of direction.



Only the NTP server port number is fixed.  The protocol (v3 and v4, at least) and reference implementation handles random client port numbers just fine, same as DNS.  My public stratum 2 server sees around half of its clients using random source ports, currently 5,701 using port 123 out of 10,460 clients, with the random ports mostly being evenly spread across the full range other than a spike of 153 using port 1024.

Games which are strictly client<-->server have no need to use a fixed client port.  Where they also have peer to peer comms, they often use fixed UDP ports as a way to pierce NAT and stateful firewalls.  If 2 peers both start a UDP conversation simultaneously (having obtained the remote peer's address from a central server), that will open sufficient NAT and firewall states at both ends to allow them to talk to each other when random inbound UDP would normally be blocked or dropped.


----------



## Murph (Aug 10, 2016)

Murph said:


> My public stratum 2 server sees around half of its clients using random source ports, currently 5,701 using port 123 out of 10,460 clients, with the random ports mostly being evenly spread across the full range other than a spike of 153 using port 1024.



Oops, wrong query used there, that was only the more active recent clients (minimum 50 packets).  Looking at all recent clients, 8,019 out of 42,091 use port 123, with the remainder spread fairly evenly around the range (and spikes of 273 on port 1024, 72 on 32769, 56 on 2052, and 42 on 1025; with smaller spikes giving strong evidence of sequential port allocation starting at bases of 1024, 2048, 3072, 32768, and 49152; and evidence of a subset of the population restricting their random port choice to the official high range of 49152–65535 and others to the upper half of 32768–65535).


----------



## fossette (Aug 10, 2016)

Testing a random number generator is very simple.  No need to use the whole TCP stack, but it's the utmost importance to find out exactly which routine is being used.  Testing the wrong routine won't bring any good.

What I do is simply brute force the algorithm.  That is, pick a seed at the beginning, then repetitively call the routine to generate random numbers and log them in memory.  What we are looking for is when the pattern repeat itself.  The longer it is, the better, so it's more difficult to predict.  Note that this sequence is the result of the original seed, and the random mathematical function algorithm.

If the log shows the same value that repeats itself, then the mathematical function is less than par.  This is why weak random number generator algorithms require that the seed be regularly refreshed by the timeclock (or some other external event like a key pressed on the keyboard) for the repeat pattern or stuck sequence to be less predictable.  For telecommunication, encryption, or a casino slot machine, this is a vulnerability.

No random number generator is perfect.  This is why we call them Pseudo Random Number Generators.

Dominique.


----------



## Murph (Aug 10, 2016)

You really need to look at more than just when it repeats.  Ideally you want to be checking for a reasonably flat distribution across the range; possibly via the standard deviation, or just counting the results into buckets and confirming that all buckets are receiving a similar number of data points.


----------



## ronaldlees (Aug 10, 2016)

Murph said:


> Ahh, here we are it is documented in TFM, ip(4):



That clarifies the port picking quite a bit. But, by those terms, my machines would have always been in "random" mode.


----------



## ronaldlees (Aug 10, 2016)

Murph said:


> You really need to look at more than just when it repeats.  Ideally you want to be checking for a reasonably flat distribution across the range; possibly via the standard deviation, or just counting the results into buckets and confirming that all buckets are receiving a similar number of data points.



Right, because that validates the strength of the algorithm and its entropy, rather than a datum from a point in time (or several).


----------



## fossette (Aug 10, 2016)

Murph said:


> Ideally you want to be checking for a reasonably flat distribution across the range


Murph, with that assumption, one could expect to win the lottery just by picking the same combination ticket week after week.  Lol!!!
Dominique.


----------



## Murph (Aug 10, 2016)

ronaldlees said:


> That clarifies the port picking quite a bit. But, by those terms, my machines would have always been in "random" mode.



DNS activity could, perhaps, be generating frequent spikes of port allocations.  I'd need to do some digging into source to confirm it, but I think BIND will allocate new random ports (via the OS allocator) for each query that it generates.  Unbound is probably similar.  With several levels of delegation, and corresponding DNSSEC queries, repeating truncated queries via TCP, and querying both IPv4 and IPv6 addresses, resolving a single hostname to address can easily involve quite a few DNS queries.


----------



## ronaldlees (Aug 10, 2016)

fossette said:


> Murph, with that assumption, one could expect to win the lottery just by picking the same combination ticket week after week.  Lol!!!
> Dominique.



If you didn't have a flat distribution, then indeed you could expect more value from your carefully chosen lottery ticket!


----------



## Murph (Aug 10, 2016)

fossette said:


> Murph, with that assumption, one could expect to win the lottery just by picking the same combination ticket week after week.  Lol!!!
> Dominique.



Not really.  If you only look for repeats, `i++;` would pass the test until integer overflow.  Verifying that the distribution is reasonably flat is essential.  Lottery numbers should be a flat distribution, otherwise you could look at past statistics and improve your chances by picking numbers in the hotter part of the distribution.  Lotteries rely on the very large number of possible results, combined with a very flat distribution.


----------



## ronaldlees (Aug 10, 2016)

Murph said:


> DNS activity could, perhaps, be generating frequent spikes of port allocations.  I'd need to do some digging into source to confirm it, but I think BIND will allocate new random ports (via the OS allocator) for each query that it generates.  Unbound is probably similar.  With several levels of delegation, and corresponding DNSSEC queries, repeating truncated queries via TCP, and querying both IPv4 and IPv6 addresses, resolving a single hostname to address can easily involve quite a few DNS queries.



Maybe I'll keep one machine around just to see if I can duplicate this scenario.  Nothing seems amiss today.  The "mode switching" described in the ip() man page means much more involvement within the allocator than simple use of a PRNG - so we shouldn't necessarily blame the latter or blame anything OS related.  I'm still leaning  towards the stray bit leaking through my firewall as the origin of this thing.  It's definitely not normal.

Update: It's two weeks later, and still there are no further signs of any ports being stuck to #10000.  Whatever it happened to be, it is/was an infrequent thing. Glad for that.  One other curious item that I didn't mention:  The ISP DNS here was excruciatingly slow for about three days in the referenced time frame.  It was so slow that I eventually changed the config on everything so it would get DNS from the outside (ISP was averaging over 2 seconds per query, and as long as 5 seconds).


----------

