# sshd - security run output



## frabron (Jun 22, 2012)

Hi,

I really appreciate the daily mails you get from the system. One thing that bothers me is the output included from the sshd service in the daily security output mail. There are always plenty of lines like

```
...
Jun 21 15:54:21 frodo sshd[86804]: Invalid user u44628 from 205.251.131.180
Jun 21 15:54:23 frodo sshd[86808]: Invalid user coferus from 205.251.131.180
Jun 21 15:54:47 frodo sshd[86857]: Invalid user allsystems from 205.251.131.180
Jun 21 15:54:53 frodo sshd[86869]: Invalid user dolphy from 205.251.131.180
...
```
which is neither too interesting nor very helpful since this is just background noise. Today I had 64 lines like the ones quoted. You always have to scroll very much and the long list distracts from the other security messages. So, is there a way to achieve an output like

```
Jun 21: 64 invalid user login attempts
Jun 21: 3 failed logins
```
a summary of failed logins? I can always look myself at /var/log/auth.log if I'm really interested. That would me more helpful than listing all the dictionary attacks in the email.

I'd prefer not to disable the sshd security information completely but a more terse output would be nice.

Frank


----------



## SirDice (Jun 22, 2012)

Install something like security/sshguard. It'll help protect against all those brute-force attempts.


----------



## frabron (Jun 22, 2012)

Thanks for the tip, but this is not the issue of my concern. I don't mind those brute-force attempts. It's really just the email formatting. But I'll have a look at sshguard


----------



## SirDice (Jun 22, 2012)

I think I understand the issue, the email can get quite long if you have hundreds of failed attempts. It would be nice if it only showed the offending lines if there are, lets say, 10 failed attempts. If there are more it should just say "64 invalid users and 12 failed attempts" for example.

The report is created by the /etc/periodic/security/800.loginfail script. The functionality you want isn't there though. It would require a bit of coding but the idea is good.


----------



## frabron (Jun 22, 2012)

Thanks, that's what I am looking for (the script at least ). I will polish my shell fu a little bit and see what I can do. If I create another file 801.loginfail for testing purposes would the output get integrated into the security mail?


----------



## SirDice (Jun 22, 2012)

frabron said:
			
		

> If I create another file 801.loginfail for testing purposes would the output get integrated into the security mail?


Yes, it would.


----------



## frabron (Jun 22, 2012)

Ok, I made up a script that should do what I want. Here's the modified part:

/etc/periodic/security/800.loginfail

```
case "$daily_status_security_loginfail_enable" in
    [Yy][Ee][Ss])
        echo ""
        echo "${host} login failure summary:"

        # number of failed logins
        num_failed=$(catmsgs | egrep -ia "^$yesterday.*: .*(fail)" | wc -l)
        echo "$yesterday had $num_failed failed login attempts" | tee /dev/stderr

        # number of invalid logins
        num_invalid=$(catmsgs | egrep -ia "^$yesterday.*: .*(invalid)" | wc -l)
        echo "$yesterday had $num_invalid invalid login attempts" | tee /dev/stderr
 
        # number of bad login attempts
        num_bad=$(catmsgs | egrep -ia "^$yesterday.*: .*(bad)" | wc -l)
        echo "$yesterday had $num_bad bad login attempts" | tee /dev/stderr

        # number of illegal login attempts
        num_illegal=$(catmsgs | egrep -ia "^$yesterday.*: .*(illegal)" | wc -l)
        echo "$yesterday had $num_illegal illegal login attempts" | tee /dev/stderr
        ;;
#       original statements
#       n=$(catmsgs | egrep -ia "^$yesterday.*: .*(fail|invalid|bad|illegal)" |
#           tee /dev/stderr | wc -l)
#       [ $n -gt 0 ] && rc=1 || rc=0;;
    *)  rc=0;;
esac
```

I am by no means a good shell user, so my attempts may look funny or worse for anyone with more knowlegde in shell programming than me (which are propably a lot of people :e)
I am still concerned about the rc line in the original script:

```
[ $n -gt 0 ] && rc=1 || rc=0;;
```
*A*s far as I understand this line, it tests if $n (the number of messages in total) is greater than 0 and if the rc level is 1, and if that evaluates to false set the rc level to 0. So I don't know if I need that or not. Suggestions are welcome 

*C*alling `# sh 800.loginfail` yields:

```
login failure summary:
Jun 21  had        3 failed login attempts
Jun 21  had        3 failed login attempts
Jun 21  had       61 invalid login attempts
Jun 21  had       61 invalid login attempts
Jun 21  had        2 bad login attempts
Jun 21  had        2 bad login attempts
Jun 21  had        0 illegal login attempts
Jun 21  had        0 illegal login attempts
```

There are some variables missing due to the different context in which the script is normally called I guess, but I will give it a try like this and see what's happening.

Frank


----------



## SirDice (Jun 22, 2012)

frabron said:
			
		

> I am still concerned about the rc line in the original script:
> 
> ```
> [ $n -gt 0 ] && rc=1 || rc=0;;
> ...


It's a bit tricky to read but it basically does the same thing as this bit of pseudocode:

```
if $n > 0 then
   rc=1
else
   rc=0
```

So depending on the value of $n the variable rc gets set to 0 or 1.


----------



## frabron (Jun 25, 2012)

FYI: I reworked the script a little bit to eliminate the doubled line output (useless use of tee ) and enabled the rc variable handling again.


```
case "$daily_status_security_loginfail_enable" in
    [Yy][Ee][Ss])
        echo ""
        echo "${host} login failure summary:"

        # number of failed logins
        num_failed=$(catmsgs | egrep -ia "^$yesterday.*: .*(fail)" | wc -l)
        echo "$yesterday had $num_failed failed login attempts" > /dev/stderr

        # number of invalid logins
        num_invalid=$(catmsgs | egrep -ia "^$yesterday.*: .*(invalid)" | wc -l)
        echo "$yesterday had $num_invalid invalid login attempts" > /dev/stderr

        # number of bad login attempts
        num_bad=$(catmsgs | egrep -ia "^$yesterday.*: .*(bad)" | wc -l)
        echo "$yesterday had $num_bad bad login attempts" > /dev/stderr

        # number of illegal login attempts
        num_illegal=$(catmsgs | egrep -ia "^$yesterday.*: .*(illegal)" | wc -l)
        echo "$yesterday had $num_illegal illegal login attempts" > /dev/stderr

        if [ $num_failed > 0 -o $num_invalid > 0 -o $num_bad > 0 -o $num_illegal > 0 ]
        then
            rc=1
        else
            rc=0
        fi;;

    *)  rc=0;;
esac
```


```
login failure summary:
Jun 24  had       41 failed login attempts
Jun 24  had       59 invalid login attempts
Jun 24  had        2 bad login attempts
Jun 24  had        0 illegal login attempts
```


----------



## anomie (Jun 25, 2012)

Thank you for sharing your changes. If you're accepting code suggestions, you might consider replacing the pipeline to wc(1) with a simple addition to your egrep(1) options instead. 


```
-c, --count
              Suppress  normal output; instead print a count of matching lines
              for each input file.
```


----------



## vadipp (Jul 15, 2013)

I want to post my version of the same thing. I had to rewrite it a bit, because @frabron's version doesn't catch "authentication error" kind of errors which I believe are the most crucial. I could start worrying if someone tries to hack into an existing user, but usually don't care about non-existent ones. I've also simplified the script a bit. So here it goes:

```
case "$daily_status_security_loginfail_enable" in
    [Yy][Ee][Ss])
        echo ""
        echo "${host} login failures:"
        rc=0
        for kind in "Invalid user" "not allowed" "authentication error" "BREAK-IN"; do
            num_kind=$(catmsgs | egrep -ac "^$yesterday.*: .*$kind ")
            if [ $num_kind -gt 0 ]; then
                echo "$yesterday had $num_kind $kind login attempts"
                rc=1
            fi
        done
esac
```
This works fine on 9.1-RELEASE.

Sample output:

```
login failures:
Jul 14  had 173 Invalid user login attempts
Jul 14  had 1 authentication error login attempts
Jul 14  had 689 BREAK-IN login attempts
```


----------



## throAU (Jul 16, 2013)

If you have large amounts of attempts hitting your SSH port, the more important thing to do is be blocking SSH inbound unless it's from a trusted source (e.g., preferably IPsec tunnel, VPN or dedicated management network), or at least a network you're likely to be connecting from - i.e., the netblock from your ISP).

There is zero reason to be listening for SSH connections from ISPs (or even entire countries) where you have no staff, for example.

Reducing the security log spam by modifying the report is merely masking the actual problem in my opinion - you want to get those failed login attempts down to a number you can actually investigate.

Also - if you're not using public/private key authentication (and have password authentication for SSH turned off), look into it.


----------



## vadipp (Jul 18, 2013)

That's generally true, but it's my personal server, and I want to be able to easily connect to it from anywhere using plain SSH to access my data. So it doesn't make sense for me to restrict SSH connections. I just want to see if someone is trying to hack into root or any other existing user on the server. I think I'm not alone in this situation, but I encourage everyone to make their own security decisions.


----------



## throAU (Jul 18, 2013)

vadipp said:
			
		

> That's generally true, but it's my personal server, and I want to be able to easily connect to it from anywhere using plain SSH to access my data. So it doesn't make sense for me to restrict SSH connections. I just want to see if someone is trying to hack into root or any other existing user on the server. I think I'm not alone in this situation, but I encourage everyone to make their own security decisions.



If this is the case and securing behind a VPN is not feasible, I'd still definitely recommend setting up log in via SSH keys.

The need for an overly complex password is significantly lessened by using public/private key authentication.

Public/private key authorization may seem scary at first, but once you've done it a few times it will become familiar - and it is MUCH more secure if used properly (i.e., with a passphrase on your private key).

In case you're not aware of how to do it, this is the process (brief overview):

On your personal non-exposed machine (e.g., your personal laptop, workstation, etc.), run ssh-keygen to generate a public/private keypair.  This will give you 2 files, depending on which options you select (v1 vs. v2, DSA vs RSA key type), will be ~/.ssh/identity and ~/.ssh/identity.pub, or id_rsa and id_rsa.pub, or id_dsa and id_dsa.pub.

In any case, the .pub file is your "public" key, and the contents of this file should be added to ~/.ssh/authorized_keys on any remote machine you want to access (either cut/paste using a text editor or `cat identity.pub >> ~/.ssh/authorized_keys`).  The corresponding "identity" file (e.g., the filename without .pub on the end) is your private key, and should be protected - kept on a private, secure machine and not disclosed to anyone, ever.  This key can optionally be password protected (in case someone manages to steal your private key, they still need to enter a password to decrypt it) as well.

Once you have the public key on your remote machine in ~/.ssh/authorized_keys, you should be able to connect via key authentication and can then turn off password authentication.  If you have a password on your key, you'll still need to enter the password when using the key, but this traffic is never sent over the wire, key decryption using your key's passphrase is done locally before the connection is made.

Once you've done this, for someone to break in via ssh they will need to crack (or steal) your private key, which is much, much more difficult than a dictionary attack on your password (i.e., 1024 bits of entropy or more, vs say 110-120 bits of entropy for a fully random 16 character password).  IF someone does manage to steal your private key, the passphrase will prevent them from using it in the time frame between you having it stolen and realizing that you've had it stolen, and replacing it. A known or suspected stolen key should be removed from authorized_keys on all machines and a new public/private key pair generated, in case someone who has the key manages to brute-force your passphrase on the key.  If your key has no passphrase, anyone who has it can log in.

Step by step process available here:  http://www.petefreitag.com/item/532.cfm (not my guide).  Only difference is that FreeBSD uses "authorized_keys" and not "authorized_keys2".  Also the guide doesn't specify using a passphrase (ssh-keygen *will* prompt you for one, however).  USE a passphrase on your key.  This will prevent someone who steals your key from being able to use it.

You can even reduce the need to keep entering your passphrase with ssh keys by using a program like `ssh-agent`, or if you are a Windows user, Putty Agent.


----------



## ta0kira (Jul 18, 2013)

vadipp said:
			
		

> That's generally true, but it's my personal server, and I want to be able to easily connect to it from anywhere using plain SSH to access my data. So it doesn't make sense for me to restrict SSH connections. I just want to see if someone is trying to hack into root or any other existing user on the server. I think I'm not alone in this situation, but I encourage everyone to make their own security decisions.


It _does_ make sense, if you recognize that a lot of those break-in attempts are for the purposes of gathering intelligence, or with the objective of using your machine to host attacks on other systems. When my firewall is completely open I get thousands of login failures per night from single IP addresses. It's not just "someone" trying to break in, but rather organizations who employ people to break into systems.

If you're going to leave port 22 open to the entire planet, you can at least restrict wheel logins to a certain subset of IP addresses in /etc/login.access. I have two accounts, one with wheel and one without. My firewall limits access to a small subset of IP addresses, and I restrict wheel-member logins to an even smaller subset.

Kevin Barry


----------



## Savagedlight (Jul 19, 2013)

Lots of good stuff in here.

Is there any way to make this script attach the relevant raw logs as an attachment to the security output e-mail?


----------



## Chris_H (Jul 19, 2013)

Sure. Just add it to periodic(8). 

--chris


----------



## Savagedlight (Jul 19, 2013)

That would make the script main output end up inside the main e-mail body. 

I was wondering if there was any way to add the detailed (non-summarized) output as a .txt attachement, in case studying the details would be interesting for some reason.


----------



## Chris_H (Jul 19, 2013)

Savagedlight said:
			
		

> That would make the script main output end up inside the main e-mail body.
> 
> I was wondering if there was any way to add the detailed (non-summarized) output as a .txt attachement, in case studying the details would be interesting for some reason.



Ahhh, I see. I was thinking you simply wanted the output, as with the other security emails. So simply naming it something like: 801.sshfail, and dumping it in /etc/periodic/security would get it. But as for an attachment; why not simply add a cron(8) job that (e)greps the auth.log just before periodic(8) picks it up, sending it's output somewhere, maybe even as a tarball. Then append something to 800.loginfail that appends the file you saved earlier.

--chris


----------



## pacija (Nov 22, 2013)

I am getting security run output from a server which always contains information about failed SSH logins. However I don't see any records in auth.log. Where does it get information from then?


----------



## frabron (Nov 25, 2013)

Well, it should be in there. Did you have a look at the archived log files? The login failures from the latest email are usually archived, since they're from yesterday, in the auth.log.0.bz2 file. You can have a quick look with zless, instead of unpacking and less.


----------

