# System Compromised!



## shahzaib (Feb 26, 2016)

Hi,

A user systemd was created with root privileged on FreeBSD, once we started digging more into this we found the .bash_history file under his home directory which he created in /lib/.systemd, his history explained that he downloaded some log tamper script in order to hide his appearance but fortunately he wasn't able to install it (some binary error occurred). What more interesting is, generally when you create a user via root the following log is created in userlog :

```
userlog:2016-02-25 22:36:55 [[B]root[/B]:useradd] systemd(6) home /myuser made
```
But when we checked about the creation log of that mysterious 'systemd' user, this is what we found :

```
userlog:2016-02-25 22:36:55 [[B]unknown[/B]:useradd] systemd(6):wheel(0):system daemon:/lib/.systemd:/usr/local/bin/bash
userlog:2016-02-25 22:36:55 [[B]unknown[/B]:useradd] systemd(6) home /lib/.systemd made
```

So systemd was created by some unknown user which doesn't exist anywhere in system. We also thought it might be privilege escalation OpenSSH vulnerability to gain root access,so here is the ssh version but we failed to find any known vulnerability about this version :

```
OpenSSH_6.6.1p1, OpenSSL 1.0.1j-freebsd 15 Oct 2014
```
Here is the FreeBSD version :

```
10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401
```
If you guys can help me tracing the issue, it'd be very helpful.


----------



## Juha Nurmela (Feb 26, 2016)

pw(8) uses "unknown" when LOGNAME and USER are not set. 

Disturbing sprintf(3) in pw/pw_log.c,
Juha


----------



## kpa (Feb 26, 2016)

What is this system running, any PHP based web services? Give us more information.


----------



## Oko (Feb 27, 2016)

shahzaib said:


> So systemd was created by some unknown user which doesn't exist anywhere in system. We also thought it might be privilege escalation OpenSSH vulnerability to gain root access,so here is the ssh version but we failed to find any known vulnerability about this version :
> 
> ```
> OpenSSH_6.6.1p1, OpenSSL 1.0.1j-freebsd 15 Oct 2014
> ...



You are mixing OpenSSH and OpenSSL. OpenSSL which has nothing to do with OpenBSD had a major vulnerability problem. OpenSSH no longer uses libcrypto which was a  part of OpenSSL. FreeBSD version of OpenSSH has been upgraded long time ago. As a mater of fact OpenBSD doesn't use OpenSSL period unlike FreeBSD. The project uses its own LibreSSL which is about 50% smaller code base than OpenSSL. My understanding is that FreeBSD is completely moving to LibreSSL as well. The real puzzle to me is why are you using outdated/obsolete software? Any person who cares about security does regular upgrades and patches.


----------



## shahzaib (Feb 27, 2016)

Oko said:


> You are mixing OpenSSH and OpenSSL. OpenSSL which has nothing to do with OpenBSD had a major vulnerability problem. OpenSSH no longer uses libcrypto which was a  part of OpenSSL. FreeBSD version of OpenSSH has been upgraded long time ago. As a mater of fact OpenBSD doesn't use OpenSSL period unlike FreeBSD. The project uses its own LibreSSL which is about 50% smaller code base than OpenSSL. My understanding is that FreeBSD is completely moving to LibreSSL as well. The real puzzle to me is why are you using outdated/obsolete software? Any person who cares about security does regular upgrades and patches.



Well, we recently shifted to FreeBSD (2-3 months back) and that was the default ssh we found, could you please let me know how to upgrade to accurate FreeBSD SSH ?


----------



## shahzaib (Feb 27, 2016)

kpa said:


> What is this system running, any PHP based web services? Give us more information.


Yes, its nginx+php-fpm which serves php as well.


----------



## scottro (Feb 27, 2016)

Idle curiosity--I wonder if the reason they were unable to install the log tamper script is because it was aimed at Linux. The fact that the user is called systemd, which is the name of the primary process on most recent Linux distributions, points in that direction.


----------



## kpa (Feb 27, 2016)

shahzaib said:


> Yes, its nginx+php-fpm which serves php as well.



I would first look here for the vulnerability/backdoor although your system has no security patches at all for 10.1 (the SVN revision reveals that) and that also puts it at great risk.


----------



## Oko (Feb 27, 2016)

FreeBSD is soon to be 10.3 and you are using 2 year old 10.1 At least upgrade to the latest stable 10.2 and then post.


----------



## kpa (Feb 27, 2016)

Oko said:


> FreeBSD is soon to be 10.3 and you are using 2 year old 10.1 At least upgrade to the latest stable 10.2 and then post.



Actually there's no need to upgrade yet,  the support period of 10.1 lasts till 31st of December, 2016.


----------



## atomicbeef (Feb 29, 2016)

As Oko said before, it is important to keep your system up to date. In FreeBSD, this means keeping both your ports/packages and the base system up to date. To keep the base system up to date, see freebsd-update(8).


----------



## Oko (Feb 29, 2016)

kpa said:


> Actually there's no need to upgrade yet,  the support period of 10.1 lasts till 31st of December, 2016.


So you are suggesting him to stick with "supported" 10.1 branch full of security problems. Of course I am exaggerating as I see now advantage of sticking to updated version of 10.1 comparing to upgrading to 10.2. In particular I use ZFS for the root and `beadm` is my safety valve.


----------



## jrm@ (Feb 29, 2016)

It's probably a moot point now.  After discovering how the system was penetrated (I think) it's best to wipe it clean and start fresh.  Might as well go with the latest release at that point.


----------



## gkontos (Feb 29, 2016)

shahzaib said:


> Well, we recently shifted to FreeBSD (2-3 months back) and that was the default ssh we found, could you please let me know how to upgrade to accurate FreeBSD SSH ?


You can use freebsd-update() to get to the latest 10.1-RELEASE patchset which includes various openssl bug fixes. Like kpa said, there is no need to upgrade to a more recent version as your version is still supported. However, like he also mentioned, you should audit your web application for vulnerabilities.


----------



## Crivens (Feb 29, 2016)

And before wiping the system I would suggest to image the partitions which could contain information from the breakin. You may think you know how someone got in, but you should prefer to _know_ how that came to pass.


----------



## jrm@ (Feb 29, 2016)

gkontos said:


> You can use freebsd-update() to get to the latest 10.1-RELEASE patchset which includes various openssl bug fixes. Like kpa said, there is no need to upgrade to a more recent version as your version is still supported.


The concern I would have in such a scenario would be never really knowing _all_ that has been touched.  Once an intruder has root, it's game over.


----------



## gkontos (Mar 1, 2016)

jrm said:


> The concern I would have in such a scenario would be never really knowing _all_ that has been touched.  Once an intruder has root, it's game over.


Absolutely, the system should be reinstalled. However, if you noticed I also recommended to audit the web application for vulnerabilities. If they just reinstall the same thing might happen again.


----------



## SirDice (Mar 1, 2016)

There might be another scenario at play here. As far as I've been able to find there are no local or remote root exploits for (an unpatched) 10.1. That basically rules out the attackers elevating their privileges through a vulnerable website. What's more likely is someone brute-forcing their way into an account through SSH. If that account also has full sudo(8) access they would have all the access they need. 

I'd take a very long and hard look at /var/log/auth.log.


----------



## atomicbeef (Mar 1, 2016)

SirDice said:


> There might be another scenario at play here. As far as I've been able to find there are no local or remote root exploits for (an unpatched) 10.1. That basically rules out the attackers elevating their privileges through a vulnerable website. What's more likely is someone brute-forcing their way into an account through SSH. If that account also has full sudo(8) access they would have all the access they need.
> 
> I'd take a very long and hard look at /var/log/auth.log.


You should definitely do this (assuming that you haven't already reinstalled). If someone was brute forcing their way into an account, you should see a bunch of messages about them failing to authenticate to the user and their IP. Eventually in the log you'll find that the same IP that was failing to log in managed to get into your system.

If you were the victim of a brute force attack, the first thing you should do is come up with a stronger password. You should also look into Intrusion Prevention Systems such as fail2ban(1) to prevent brute force attacks from happening in the first place.


----------



## gkontos (Mar 2, 2016)

SirDice said:


> There might be another scenario at play here. As far as I've been able to find there are no local or remote root exploits for (an unpatched) 10.1. That basically rules out the attackers elevating their privileges through a vulnerable website. What's more likely is someone brute-forcing their way into an account through SSH. If that account also has full sudo(8) access they would have all the access they need.
> 
> I'd take a very long and hard look at /var/log/auth.log.


The logs indicate that the user was created by none. I think that if an account had been brute forced, then the correct username would appear. 

It would help if we knew what user was used to run nginx and/or if the application can modify system files.


----------



## lme@ (Mar 2, 2016)

As the other said. Upgrading the system to the latest 10.1 patch release would not be wise. It's like a burglar is in your house because your door lock was too simple. Now you replace it with a super-secure one, but the burglar is still in your house, rummaging through your laundry.


----------



## Crivens (Mar 2, 2016)

Juha Nurmela said:


> pw(8) uses "unknown" when LOGNAME and USER are not set.





gkontos said:


> The logs indicate that the user was created by none. I think that if an account had been brute forced, then the correct username would appear.


Maybe if you unset the names from the environment, then this is also the outcome? It would be trivial, and remove a bit of information from the log files.


----------



## Juha Nurmela (Mar 2, 2016)

it should use getlogin(2) to get the name and getlogin would be nice, if it raised the LAAAAAAARM when there is no login name yet.

Juha


----------

