# Recent botnet installed as root



## ernie (Mar 22, 2012)

I just had 3 machines 6.2-STABLE, 6.3-STABLE, 7.2-RELEASE, all infected on the same day with a botnet. In the /tmp dir there were two abnormal files

```
-rw-r--r--   1 root   wheel     0B Mar 22 15:58 null
-r-xr-xr-x   1 root   wheel   393K Mar 22 07:05 talkngbsd
```

MD5 (talkngbsd) = 5b2744053752a93462d6fec7c2347c24

There were multiple instances of talkngbsd running, and it was IP spoofing and attacking an overseas server on port 80, no doubt trying to do a DOS. I have no idea how the program got installed as root, there are no user accounts on any of the machines, no ssh enabled, the only thing they have in common is Apache and sendmail running. Are there any know exploits for those that could escalate privileges?


----------



## kpa (Mar 22, 2012)

What version of www/apache port? All of the FreeBSD versions that you mention have been end of life for a while meaning there are no security updates for them even if serious security holes are found. Clean up the mess and upgrade to at least 7.4 or preferably 8.2.


----------



## ernie (Mar 22, 2012)

/var/db/pkg/apache-2.2.4_2 on two of them but I just realized the third machine didn't have Apache installed, so that kind of rules out apache. 

From rc.conf the processes that were common to all 3


```
bsnmpd_enable="YES"
gateway_enable="YES"
inetd_enable="YES"
ntpdate_enable="YES"
sendmail_enable="YES"
usbd_enable="YES"
```


----------



## xibo (Mar 22, 2012)

*W*hat is enabled via inetd?


----------



## ernie (Mar 22, 2012)

xibo said:
			
		

> what is enabled via inetd?




```
#
#ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l
#telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd
login   stream  tcp     nowait  root    /usr/libexec/rlogind    rlogind
```

Telnet and ftp are usually disabled and activated as needed, I normally login with rlogin, the su password is quite complex.


----------



## kpa (Mar 22, 2012)

> ```
> login stream tcp nowait root /usr/libexec/rlogind rlogind
> ```



Is that open directly to the internet? You're asking for a lot of problems if it is.


----------



## redw0lfx (Mar 22, 2012)

ernie said:
			
		

> #
> #ftp    stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l
> #telnet stream  tcp     nowait  root    /usr/libexec/telnetd    telnetd
> login   stream  tcp     nowait  root    /usr/libexec/rlogind    rlogind
> ...



I would suggest using OpenSSH instead of rlogin.  Also, you are running Apache 2.2.4 which has some security vulnerabilities. Current version of Apache is 2.2.22 (unless you have kept it patched).  If they wrote to /tmp, I wouldn't doubt they came in through apache specially if mod_php is enabled.  However, rlogin could have unknown exploits as well.  

I would look at the apache logs to see if any weird urls were called, mainly stuff with \x0a\x78\x... etc.  I would also check history on the root user and perhaps your authentication logs.  However, keep in mind that they could have cleaned up after themselves.



			
				ernie said:
			
		

> I have no idea how the program got installed as root, there are no user accounts on any of the machines, no ssh enabled, the only thing they have in common is Apache and sendmail running. Are there any know exploits for those that could escalate privileges?



You have to look at what services are exposed to the internet, and determine if there are any known exploits for them that would give at least a shell prompt.  Once you have a shell prompt, all bets are off, as you can then attempt to exploit any installed binary to escalate privileges, for example, through mutt, vi, etc.  

Once you are satisfied no more information can be obtained from the compromised system, you should re-install your FreeBSD system with a current version, as who knows what else they could have done.


----------



## ernie (Mar 22, 2012)

I think it might have been telnetd, I hadn't noticed the security advisory from Dec 2011, which is odd as I am on the mail list. I use rlogind only internally, ssh fills the logs with probe attempts. Upgrading the 7.2-RELEASE machine to 7.4-RELEASE will be easy, the 6.2-STABLE, 6.3-STABLE machines I am not quite sure how to go about. Welcome any suggestions.


----------



## jem (Mar 23, 2012)

Having your logs fill with ssh probe attempts is a lesser evil than having your boxes rooted. It's also easily avoided by running ssh on a non-standard port.

Not using plain text protocols like rlogin to log into your systems is the absolute number one security measure that most system admins should follow.  That's been pretty standard for many years.


----------



## redw0lfx (Mar 23, 2012)

Ah, yes, if telnetd is exposed to the internet it is also a possiblity.  As for upgrading the 7.2 system to 7.4, do you really trust that you found all rootkits/botnets installed on the system and removed them, as well as blocked any backdoors they might have setup?


----------



## ernie (Mar 23, 2012)

redw0lfx said:
			
		

> Ah, yes, if telnetd is exposed to the internet it is also a possiblity.  As for upgrading the 7.2 system to 7.4, do you really trust that you found all rootkits/botnets installed on the system and removed them, as well as blocked any backdoors they might have setup?



I find that when you do an upgrade, any hacked binaries get overwritten, mergemaster should point out any diffs. I discovered that telnetd had been deleted by the hacker, on the 3 machines concerned, I can't think why, perhaps the bot binary intercepts port 23? I will run it up on an isolated test network and see what ports it opens. Two of the 3 machines were hacked within seonds of each other, I suspect it was automated.


----------



## SirDice (Mar 23, 2012)

jem said:
			
		

> Having your logs fill with ssh probe attempts is a lesser evil than having your boxes rooted.


Just use something like security/sshguard.



> Not using plain text protocols like rlogin to log into your systems is the absolute number one security measure that most system admins should follow.  That's been pretty standard for many years.


I fully agree. Don't use telnet or rlogin. That's asking for trouble. Even if you only use it 'internally'.

It quite possible they came in through a web application, started sniffing and grabbed the root password you typed in on an insecure connection. Once root was obtained they had full reign over your servers.

Backup your data and wipe everything. Reinstall using a current and supported version.


----------



## SirDice (Mar 23, 2012)

ernie said:
			
		

> I discovered that telnetd had been deleted by the hacker, on the 3 machines concerned, I can't think why, perhaps the bot binary intercepts port 23?


No, it's to prevent others from getting in through the same hole. They don't like it when some other hacker takes over their pwn3d servers.


----------



## SirDice (Mar 23, 2012)

ernie said:
			
		

> Telnet and ftp are usually disabled and activated as needed, I normally login with rlogin, the su password is quite complex.


Complexity of the password is irrelevant as it travels the wire in the clear. Anybody with a sniffer can grab it. As noted by others use ssh(1).


----------



## kleinart (Mar 27, 2012)

*attached are some 'logs' from my server*


```
MD5 (talkngbsd) = 5b2744053752a93462d6fec7c2347c24

MD5 (dnsbang4b) = 33787e6319263a2cee2b5c1fdc11c63d

MD5 (dnskill4bsd) = 72a54adf0ce791b52781aad95c20333c
  
MD5 (ro.bang) = fe291b5bbe47b10b14ca3193a83a2d1e
```


----------



## kleinart (Mar 27, 2012)

Your description of the attack matches my case as well.  I haven't yet found another report/post match, so I wanted to share my findings/factsâ€¦.

At about 2012-03-22 17:45:45 UTC, my server started transmitting ~15Mbps of spoofed UDP/DNS traffic.
    UDP any :>1024  -->  any:53

I have some NetFlow data, just a small captureâ€¦ I didn't do much analysis on the destination (anyIP:53)â€¦ but it appeared randomâ€¦ not targeted at any of the 13 ish whatever root servers or anything, not that they might not have been mixed in (small sample below).  The hack caused a DoS to me, as my ISP uplink is just 10Mbps, but the edge router is running RPF + has ACL's and the spoofed traffic never hit the net.

The server was a small box running FreeBSD 7.2â€¦ installed in Sept 2009.   Other than taking on a role in my DNS architecture, after another server died, the box wasn't doing too much and being that I was only running SSH, NTP and DNS (BIND 9.4.3-P2)â€¦ I (obviously wrongly, lesson learned), assumed the box was not much of an attack vector or targetâ€¦  I wasn't patching/updating it. My GUESS is the attach was against BIND - I still need to do some research, but probably some buffer overflow exploit is my guess.  I'm not all that nix/server savvy, so not all sure where to start with that analysis actually.

I suspect the attack could have been via the TCP side of DNSâ€¦. but that is pure speculationâ€¦ only as I haven't done anything on the server, but it hasn't reoccurred (yet).  I did put some ACL's around the box so that it can only initiate TCP connections (and they are logged - the box isn't doing anything suspicious there), and NTP and DNS is allowed in from the net (and all ICMP)â€¦. so the box is sheltered from the net of any TCP SYN attempts, etc.   The ACL's have isolated the box, and I was hoping to do some analysis, but then the box had a "reboot" command issued (I assume built into the script/binary that was loaded)â€¦ and after reboot the binaries remain in the tmp folder, but none of processes are running.  The reboot happened at Fri Mar 23 11:47. UTC.


** TIA --- any comments / suggestions / help if very much welcomeâ€¦.


Fact summary:
 Infected/hacked at  2012-03-22 17:44UTC
 FreeBSD 7.2 release
 services running, SSH, telnet (firewalled from net access - only allowed locally, not used - emergency only), BIND (BIND 9.4.3-P2) and NTP
 ~ 15Mbps of 'random' spoofed UDP traffic to dest port 53 (DNS)
 binaries/scripts were running as root -- not sure how  (BIND doesn't run as root, root is not allowed remote login, etc)
 TCP established connections to 212.112.241.58:81  (I didn't have a method to capture/sniff the traffic)
 At about Mar 23 05:27:54.989 UTC I finished isolating the box (ACL's around the box, to allow future analysis on the attack) - this broke the above TCP connections.
 Server did a "reboot" on Fri Mar 23 11:47. UTC
 Box left as-is since, haven't had time to debug.

 I just noticed that sendmail was runningâ€¦ and not sure what TCP 3130 is off-hand.  TCP 25 is isolated to only localhost/127.0.0.1.  Didn't show up in nmap scans.  This is new to me.  Possibly the attack point??


----------



## SirDice (Mar 27, 2012)

kleinart said:
			
		

> binaries/scripts were running as root -- not sure how  (BIND doesn't run as root, root is not allowed remote login, etc)


One possible scenario is that they brute-forced your secure shell. Once inside they used a local exploit (or perhaps the telnetd bug) to gain root. This is why you should never take local bugs lightly.


----------



## glocke (Sep 24, 2012)

Hi,
I recently found this on a server:

```
# md5 talkngbsd
md5 talkngbsd
MD5 (talkngbsd) = 0d41ba8126893455fc9372e15e01dfcb
```


```
# ls -l
-rwxr-xr-x  1 root  wheel  401972 Sep 16 11:28 talkngbsd
```


```
# file talkngbsd
talkngbsd: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 5.0 (500043), statically linked, FreeBSD-style, stripped
```
Strings: http://pastie.org/4790447
I don't know / can't figure out which vector the attacker used, everythings just outdated, apache with php (also webmin..) is running, so I expect the usual suspects, but who knows...

```
7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 19:59:52 UTC 2008     root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
```

There is a dropbear running:

```
root     dropbear   637   4  tcp4   *:3129                *:*
```
which is not installed.

```
# ps xawwj | grep 637
root     637     1   637   637    0 Is    ??    0:00.01 csh (csh)  (dropbear)
```


```
# fstat -p 637
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     dropbear     637 root /             2 drwxr-xr-x     512  r
root     dropbear     637   wd /             2 drwxr-xr-x     512  r
root     dropbear     637 text /usr     236554 -rwxr-xr-x  570988  r
root     dropbear     637    3* internet6 stream tcp c4443000
root     dropbear     637    4* internet stream tcp c4442cb0
```

Inode 236554 is (thanks double_p 

```
236554     1152 -rwxr-xr-x    1 root             wheel              570988 Nov 29  2010 /usr/include/php/dropbear
```


```
# ls -l /usr/include/php/dropbear
-rwxr-xr-x  1 root  wheel  570988 Nov 29  2010 /usr/include/php/dropbear
```
oOo...

```
# file dropbear
dropbear: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 6.4, statically linked, FreeBSD-style, stripped
```


```
# md5 dropbear
MD5 (dropbear) = 02a9e508a758d5fcf8822f957af8fc6a
```
Strings: http://pastie.org/4790612

If anyone has more input to this, I would appreciate. Thanks.


----------



## wblock@ (Sep 24, 2012)

ernie said:
			
		

> I find that when you do an upgrade, any hacked binaries get overwritten, mergemaster should point out any diffs.



This is months old, but still needs comment.

It's not safe to trust *anything* on a compromised system.  That includes the compiler, or install(1), mergemaster(8), or any other component involved in rebuilding the kernel or world.  Here is an article that describes what Ken Thompson did many, many years ago: http://scienceblogs.com/goodmath/2007/04/15/strange-loops-dennis-ritchie-a/.

Even if you boot from another system and then reinstall over the compromised one, it is very hard to be sure that all the compromises are fixed.  That's why the general recommendation is to back up data, wipe the system, install from scratch, and then restore just data.


----------



## m6tt (Sep 30, 2012)

You should also run a bios update from the manufacturer as well as an update for any "Lights Out"/Service processor, if you're paranoid, which maybe you should be?


----------



## Crivens (Sep 30, 2012)

... because, not being paranoid does not mean that _T.H.E.Y._ are not out to get you.
There have been proof of concept hacks of keyloggers in the keyboard controllers. So the question always is, what can you trust in any case.

Good article from Ken, BTW. Definitely a good read and also something you want to have around when it is said that some anti virus software is a cure-all.


----------



## m6tt (Oct 4, 2012)

Perhaps offtopic, but to really ensure the compiler was trusted you'd have to write it yourself in assembly on an air-gapped system, and use that to bootstrap gcc's C sources. Yikes. Or is there an easier way to ensure compiler purity?


----------



## mamalos (Oct 4, 2012)

...md5, sha1, etc hash?


----------



## m6tt (Oct 5, 2012)

Of what? The output of a trusted compiler?


----------



## throAU (Oct 5, 2012)

wblock@ said:
			
		

> This is months old, but still needs comment.
> 
> It's not safe to trust *anything* on a compromised system.



This.  Nuke it from orbit, it's the only way to be sure.



Q:  why are you still running 6.x or 7.x?  Sooner or later you're going to get owned due to the lack of support for a release so old.

Rather than waiting for that to happen you should be doing your best to migrate to a supported release ASAP.


----------



## Crivens (Oct 5, 2012)

Indeed. Doing the upgrade dance over several versions is no pain at all when compared to wiping the system completely and reinstalling followed by pulling user data from backups and in the meantime apologizing to every user for the downtime.


----------



## SirDice (Oct 5, 2012)

That's why a lot of admins work off-hours and weekends.


----------



## Terry_Kennedy (Oct 6, 2012)

throAU said:
			
		

> Q:  why are you still running 6.x or 7.x?  Sooner or later you're going to get owned due to the lack of support for a release so old.


I'm running 6.4 on 3 systems (and 8-STABLE on about two dozen other systems). Two of the 6.4 systems are running extremely modified versions of RANCID, MRTG, and NAGIOS (among other things) and I've been making a half-hearted attempt to get some of my mods accepted by the upstream maintainers. Some of them will never accept the changes as they're specific to the way my company generates an all-in-one customer dashboard. So there's a fair amount of stuff which will probably need to be rewritten (or at least new-from-scratch patches created). Another example is the IPMI-over-PPP that these older boxes use. There doesn't seem to be a way to get that to work, given the change away from Kernel PPP and a bunch of changes to the uart / sio / puc driver set. The 3rd 6.4 box is dedicated to a customer who is planning on merging whatever he's doing on that box to a new 8-STABLE box.

I do have the 3 replacement systems (all Dell PowerEdge R300s) installed with 8-STABLE and a bunch of the back-end processing is happening on them. Unfortunately, some of the 3rd-party software we use has a nasty habit of rewriting URL's into dotted-quad form, so it isn't as easy as it would seem to be to move pieces of the critical software over. Then there's things like several hundred Cisco routers sending us syslog data via IP address instead of hostname (some of those routers are not controlled by us, and getting them to change their IOS config to log to a different host is next to impossible).

So, this pretty much means a hopefully-sufficient level of pre-deployment regression testing, followed by a forklift upgrade of the whole system. Which is made a bit more problematic as at the same time we're doing this, we're also moving to a new colocation site.



> Rather than waiting for that to happen you should be doing your best to migrate to a supported release ASAP.


I'd be perfecly willing to take responsibility for keeping the kernel and base patched, but what has really set me back quite a bit with ports is the intentional breakage* of the new ports build structure. While I could keep most of my 6.4 ports up-to-date that way, the new ports build structure just throws a lot of errors and quits. I could even understand ports that were tweaked for options-ng not being buildable on old boxes, but I can't even build a port that was already installed before the Great Ports Breakage and which hasn't had its port Makefile and so on changed.

* If you look at the commits for files in /usr/ports/Mk, you'll see a bunch of commits to fix backward compatibiliy, but they were backed out and not resubmitted.


----------



## SirDice (Oct 8, 2012)

Backward compatibility on the ports system for 6.x systems won't happen, there's no support for them anymore.


----------



## Never (Oct 14, 2012)

Hello Everyone.
I had the same problem. I found on my FREEBSD 2 files, "talkngbsd" and "dnskill4bsd", both installed as root / wheel. They was on /root folder. Anymore, I had several daemons working named "barbut2". if killed, they keep on working. I had to delete the 2 files, talkngbsd" and "dnskill4bsd" and then reboot. I have the 2 files i could post them somewhere 4 analysis. Now it seem to work fine, BUT I can't understand HOW DAMN they succeded. i don't have telnet working, maybe sendmail ? does anyone knows how they did the exploit?


----------



## wblock@ (Oct 14, 2012)

How old is your version of FreeBSD?  What do you have running?


----------



## throAU (Oct 15, 2012)

Terry_Kennedy said:
			
		

> I'm running 6.4 on 3 systems (and 8-STABLE on about two dozen other systems). Two of the 6.4 systems are running extremely modified versions of RANCID, MRTG, and NAGIOS (among other things) and I've been making a half-hearted attempt to get some of my mods accepted by the upstream maintainers.




Oh I get it, keeping updated is painful.  I don't like having to upgrade either.  I don't think any of us like having to mess with production boxes that are doing their job.

However, your choices are:
- backport security updates to 6.x yourself
- upgrade to a supported release
- run the gauntlet without patches and eventually get owned.

For most people, option 2 is the path of lease resistance over the long term.  Option 3 is easy over the short term, but turns into a lot of work that needs to be performed under duress eventually.

Your company needs to decide whether or not the massively customized environment you have is worth the support cost, and if so, allocate enough resources (staff) for it.


----------



## kleinart (Oct 15, 2012)

Hi Never,

I posted that same issue back in March.  I was turned off by the responses; or the lack of any constructive data/response.  I never did get an answer or insight onto the attack vector.  I am still very curious.  (My first nix hack.)  I did take an image of the box and have it somewhere.  From my understanding of hacking, I wonder if there is more value in pursuing such a database and analysis (?)... which is what I was/am seeking.

Anyway, people here seem to keep harping on the obvious... beating you up over why don't you keep patched and updated, etc, etc.  Focusing on WHY you were hacked and not HOW.  AVOIDING my question of does anyone KNOW anything about those files, the attack vector, etc??  Any such technical analysis is VERY welcome.

I just get the CLEARLY OBVIOUS... do not trust a compromised host, wipe, re-install, install latest software and keep patched.  Ok, I agree, but that was not my question.  I know I am running old code.  Other than the code, I thought the box was fairly well hardened.  I know it's good to patch, and to install the latest builds, but this was a 'hobby' box, I am not a corporation, etc, etc.  I just don't have time to keep maintaining systems all day.

I know the data exists, but I'm not sure where to start to be bluntly obvious.... I just don't have much spare time to invest now... I don't know where to find a list of all vulnerabilities for my daemons (what patches are available for me to install, etc), and possibly even cross-reference that with reports of exploits in the wild, signatures, other reports.  These files are unique... Google doesn't have a lot of matches... what are the databases / repositories of MD5's of root kits, virus, malware binaries, hacks, etc...?  And a database of 'case fies' ... I had X files, this was my system config, and if known, this is how they got in ... be it via a service, even if the exact exploit used on that service was not known, etc...???
Any hobbyists do forensics and keep such data?

I'll summarize my setup:
FreeBSD 7.2-RELEASE  (unpatched, unmaintained since install)  (release date May 1, 2009)
DNS (bind9:  $FreeBSD: src/etc/namedb/named.conf,v 1.26.2.2.4.1 2009/04/15 03:14:26 kensmith )
SSH (OpenSSH : FreeBSD-20080901)
NTP
TCP 3130  (header:  "SSH-2.0-WS-FreeBSD" ... not sure what this is...?)
Telnet (firewalled to only LAN access)

no sendmail, no Apache, root SSH access disabled

I can say that my netflow, firewall and local and remote syslog did not produce any unknown SSH logins... it's just one user (me)... so I'd say I'm 90% sure that they did not SSH into the box to load the code.


----------



## kleinart (Oct 15, 2012)

inetd.conf - only SSH & telnet v4 is enabled
(external ACL that blocks telnet to all internal hosts + syslog that sends to remote syslog all unsuccessful and successful logins via telnet and SSH ... these logs show only me getting in successfully, as do all local logs, as do all file timestamps, etc)


----------



## Terry_Kennedy (Oct 15, 2012)

throAU said:
			
		

> Oh I get it, keeping updated is painful.  I don't like having to upgrade either.  I don't think any of us like having to mess with production boxes that are doing their job.
> 
> Your company needs to decide whether or not the massively customized environment you have is worth the support cost, and if so, allocate enough resources (staff) for it.


As I mentioned, I have some two dozen boxes running 8-STABLE, which get cvsup'd (and any updated ports built) at least weekly, and which get new kernels monthly.

The big problem is the customized stuff on the three 6.x boxes. Like many businesses, we have some old code. Would it be good to re-write it from scratch? Probably. Does that make any sort of business sense at all? No.



> However, your choices are:
> - backport security updates to 6.x yourself
> - ...


My big issue here is that ports were intentionally broken and not fixed for 6.x. If you look at the commits from that time period, you'll see that there were a bunch of commit comments about "fix compatibility breakage with old infrastructure" and so on. Unfortunately, as those commits went in, the level of breakage _increased_.

I know that this is an unsupported release. However, it is very rare that FreeBSD just discards compatibility without caring. In particular, I am unaware of any kernel / ABI functions necessary for the ports infrastructure which is available in 7.x and newer but not in 6.x. So why doesn't it work? Out-of-date ruby interpreter? Something else?

It doesn't help that I'm not enamored of the new ports system, either - ports which were configured and built previously throw up a new config screen (thus halting an automated portupgrade -a). This is true even when the options haven't changed on the port - apparently this is a "feature" of the new ports options - when a port is converted to the new options structure, the user gets a configure screen again. Whether or not the values shown are the values the user selected in a prior configure process or some other set of default values seems to be at the whim of the port maintainer, or perhaps the phase of the moon.

Next, there have been changes to the new global options strings, which means that one day a port will describe the "WITH_FOO =" option with one string, and the same port built on a different day on a different system will use a completely different string to describe the same option.

The only issue I ever had with the old ports system was that upgrading some ports required a manual restart of the service, as the port build / install killed the running instance. It would be nice to have a "The following ports may need to be manually restarted:" section added to the bottom of a portupgrade -a run, listing those ports where the port's process(es) were running but were killed during the upgrade.

But I'm probably talking to an nearly-empty room about this. There was a whole ruckus when FreeBSD 4.x (IIRC) was EOL'd, causing lots of discussion and the creation of the freebsd-eol[i]@[/i]lists.freebsd.org list (which has a grand total of _two_ non-spam posts since it was created nearly 6 years ago.


----------



## throAU (Oct 15, 2012)

At the end of the day, my post wasn't about the politics involved - as far as I'm concerned as an end user, the politics are irrelevant.  It was from a pragmatic standpoint (I'm not a developer).

The situation is what it is.  Your choices are to keep up to date with a supported release, backport yourself, or get hacked.  If none of those options are suitable, then you need to consider shifting to another platform with a longer support window (again, this is me being pragmatic, I'm not a FreeBSD developer and don't speak for the FreeBSD team, etc).


As to discovering HOW it was hacked (what vector) - that isn't something anyone here on this forum can help you with really.  There are a number of holes in 6.x and probably in the third party software you also have heavily customized.


edit:
Looking at the list of software above - pretty sure there's a remote exploit for Bind from a year or two ago, let alone from 2009.

A complete list of Bind security advisories available here:
https://www.isc.org/advisories/bind


----------



## Never (Oct 15, 2012)

Hi Kleinart,
I DO agree what you say: I know that if everything is updated every world is more secure, and in this discussion I also see what to do after and not HOW they did. I think that for our nature "hacking, in good way", would be nice if we succeeded in duplicating the problem. I made some research, and I found an exploit about openSSH:  http://www.securityaegis.com/openssh-0day/ and hope to have time to try. Anyway,here the list of all packages installed. I have 6.4 FreeBSD release

```
ImageMagick-nox11-6.7.5.10 Image processing tools
OpenSSH-askpass-1.2.4.1 Graphical password applet for entering SSH passphrase
alpine-2.00_3       Mail and news client descended from Pine
apr-devrandom-gdbm-db42-1.4.5.1.3.12_1 Apache Portability Library
arc-5.21p           Create & extract files from DOS .ARC files
arj-3.10.22_4       Open-source ARJ
aspell-without-dicten-0.60.6.1_1 Spelling checker with better suggestion logic \than ispell
autoconf-2.13.000227_6 Automatically configure source code on many Un*x platfor\ms
autoconf-2.68       Automatically configure source code on many Un*x platforms
autoconf-wrapper-20101119 Wrapper script for GNU autoconf
automake-1.4.6_6    GNU Standards-compliant Makefile generator (1.4)
automake-wrapper-20101119 Wrapper script for GNU automake
bash-4.2.24_1       The GNU Project's Bourne Again SHell
bison-2.5,1         A parser generator from FSF, (mostly) compatible with Yacc
bitstream-vera-1.10_5 Bitstream Vera TrueType font collection
bsdpan-Mail-SpamAssassin-CompiledRegexps-body_0-1.0 Mail::SpamAssassin::Compile\
dRegexps::body_0 - Efficient str
ca_root_nss-3.13.4  The root certificate bundle from the Mozilla Project
cclient-2007f,1     Mark Crispin's C-client mail access routines
cdialog-1.1.20111020,1 An enhanced version of 'dialog' to work with ncurses
clamav-0.97.4       Command line virus scanner written entirely in C
compat4x-i386-5.3_9 A convenience package to install the compat4x libraries
compat5x-i386-5.4.0.8.1_1 A convenience package to install the compat5x librari\
es
compositeproto-0.4.2 Composite extension headers
cs-aspell-20040614.1_1,1 Aspell Czech dictionary
curl-7.24.0         Non-interactive tool to get files from FTP, GOPHER, HTTP(S)
cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI v\
ersion
cy-aspell-0.50.3_1,1 Aspell Welsh dictionary
cyrus-sasl-2.1.25_2 RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-saslauthd-2.1.25 SASL authentication server for cyrus-sasl2
da-aspell-1.4.42.1_1,2 Aspell Danish dictionary
damageproto-1.2.1   Damage extension headers
darts-0.32          A C++ template library that implements Double-Array
db4-4.0.14_1,1      The Berkeley DB package, revision 4
-=--:----F1  lista.txt      Top L1     (Text)-----------------------------------
For information about GNU Emacs and the GNU system, type C-h C-a.
File Edit Options Buffers Tools Help
ImageMagick-nox11-6.7.5.10 Image processing tools
OpenSSH-askpass-1.2.4.1 Graphical password applet for entering SSH passphrase
alpine-2.00_3       Mail and news client descended from Pine
apr-devrandom-gdbm-db42-1.4.5.1.3.12_1 Apache Portability Library
arc-5.21p           Create & extract files from DOS .ARC files
arj-3.10.22_4       Open-source ARJ
aspell-without-dicten-0.60.6.1_1 Spelling checker with better suggestion logic than ispell
autoconf-2.13.000227_6 Automatically configure source code on many Un*x platforms
autoconf-2.68       Automatically configure source code on many Un*x platforms
autoconf-wrapper-20101119 Wrapper script for GNU autoconf
automake-1.4.6_6    GNU Standards-compliant Makefile generator (1.4)
automake-wrapper-20101119 Wrapper script for GNU automake
bash-4.2.24_1       The GNU Project's Bourne Again SHell
bison-2.5,1         A parser generator from FSF, (mostly) compatible with Yacc
bitstream-vera-1.10_5 Bitstream Vera TrueType font collection
bsdpan-Mail-SpamAssassin-CompiledRegexps-body_0-1.0 Mail::SpamAssassin::CompiledRegexps::body_0 - Efficient str
ca_root_nss-3.13.4  The root certificate bundle from the Mozilla Project
cclient-2007f,1     Mark Crispin's C-client mail access routines
cdialog-1.1.20111020,1 An enhanced version of 'dialog' to work with ncurses
clamav-0.97.4       Command line virus scanner written entirely in C
compat4x-i386-5.3_9 A convenience package to install the compat4x libraries
compat5x-i386-5.4.0.8.1_1 A convenience package to install the compat5x libraries
compositeproto-0.4.2 Composite extension headers
cs-aspell-20040614.1_1,1 Aspell Czech dictionary
curl-7.24.0         Non-interactive tool to get files from FTP, GOPHER, HTTP(S)
cvsup-without-gui-16.1h_4 File distribution system optimized for CVS (non-GUI version
cy-aspell-0.50.3_1,1 Aspell Welsh dictionary
cyrus-sasl-2.1.25_2 RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-saslauthd-2.1.25 SASL authentication server for cyrus-sasl2
da-aspell-1.4.42.1_1,2 Aspell Danish dictionary
damageproto-1.2.1   Damage extension headers
darts-0.32          A C++ template library that implements Double-Array
db4-4.0.14_1,1      The Berkeley DB package, revision 4
db41-4.1.25_4       The Berkeley DB package, revision 4.1
db42-4.2.52_5       The Berkeley DB package, revision 4.2
dirmngr-1.1.0_8     A client for managing and downloading certificate revocatio
dmxproto-2.3.1      DMX extension headers
dovecot-1.2.17      Secure and compact IMAP and POP3 servers
e2fsprogs-libuuid-1.42.2 UUID library from e2fsprogs package
el-aspell-0.50.3_1,1 Aspell Greek dictionary
emacs-nox11-23.4_8,2 GNU editing macros
en-aspell-7.1.0     Aspell English dictionaries
encodings-1.0.4,1   X.Org Encoding fonts
es-aspell-1.11.2,1  Aspell Spanish dictionary
```


----------



## SirDice (Oct 15, 2012)

Never said:
			
		

> I have 6.4 FreeBSD release


This version has been End-of-Life since November 2010 and contains quite a number of security vulnerabilities.

If you want to know who got in and how I suggest you take it to a forensic investigator.


----------



## vertexSymphony (Oct 24, 2012)

Well, @kleinart. Post a binary or do your own analisys. Don't assume that we will explain with no binary at all (in fact, it's actually better if you give it a shot by yourself. If you don't know the tools, you could ask).

Analyzing what syscalls are used, with what, which files are touched, what is written or read, what network traffic comes up. Could be a good start.

Regards.


----------



## m6tt (Feb 20, 2013)

I do wonder.

I recently started hearing about an attack with a similar MO affecting Linux machines:
http://www.webhostingtalk.com/showthread.php?t=1235797&page=38
http://www.reddit.com/r/netsec/comments/18ro3c/sshd_rootkit/

I immediately thought of the variety of machines affected and mentioned in this thread.

If you have Linux boxes out there, you may want to check on them...seems like bad news.


----------



## throAU (Feb 20, 2013)

m6tt said:
			
		

> If you have Linux boxes out there, you may want to check on them...seems like bad news.



If you have ANY boxes (regardless of operating system) "out there", you need to be keeping them up to date.  Be it Linux, FreeBSD, Windows, etc.

Dealing with potential zero-days is bad enough; if you are running software that has had well-known vulnerabilities published for several *years* there's little point even bothering to do forensic analysis IMHO.  There are plenty of known holes; which one an attacker used isn't really relevant unless you're going to fix it and also close the other known-for-years X holes left in your un-patched machine.


The "we have so much custom stuff!" isn't really an excuse.  Yeah it sucks, but it doesn't change the reality you face.

I know posters here may not be the decision maker in the company, but it is your duty to stress to management that if they want to run some super custom solution, the cost of ownership does not stop when the software is deployed.  Resources MUST be allocated to keep the software up to date as required, or eventually you WILL get owned.  

If the company is not willing to allocate resources for this, then the software is un-supportable and should be decommissioned, unless management acknowledge the risk and accept it.  It may not be what management WANT to hear, but far better that they know about the issues involved and then make a calculated business decision ON THEIR OWN HEAD to continue running an un-supportable platform.  At the end of the day if fixing the problem is beyond the resources you have available, it is a business/financial decision to make and they will either spend the money on additional resources to mitigate the risk or they won't.

However, if you haven't alerted management to the problem, then you have essentially taken on responsibility for the security of your boxes yourself and IMHO are placing yourself personally at risk for the fall-out, should they get owned.  Don't be the fall-guy.


----------



## SirDice (Feb 20, 2013)

throAU said:
			
		

> If you have ANY boxes (regardless of operating system) "out there", you need to be keeping them up to date.  Be it Linux, FreeBSD, Windows, etc.


Couldn't agree more. Unfortunately there are a lot of people out there that think OS A is better than OS B and they don't need to secure their stuff. Because they are in the mistaken belief there's something magical called unix that will prevent any and all infections or hacks.


----------



## kpa (Feb 20, 2013)

m6tt said:
			
		

> I do wonder.
> 
> I recently started hearing about an attack with a similar MO affecting Linux machines:
> http://www.webhostingtalk.com/showthread.php?t=1235797&page=38
> ...





I tried to read trough the linked material but I couldn't find anything definite that would point to a vulnerability in sshd(8). Lots of speculation whether this is yet another PHP vulnerability though...


----------



## SirDice (Feb 20, 2013)

It's not related to sshd(8), that's just one of the symptoms. It's unclear how exactly they got in but my guess is some broken web application. They then used this 0-day Linux kernel bug to gain root and backdoor sshd(8).

http://blog.solidshellsecurity.com/2013/02/08/sshd-spam-rootkit-lib64libkeyutils-so-1-9/

Obviously just removing that library and restarting sshd(8) doesn't solve the issue it only removes the symptoms. Because the attackers had unrestricted root access nothing on that server should be trusted anymore.


----------



## m6tt (Feb 21, 2013)

I agree it's very unlikely sshd...I think the last commonly exploited remote in sshd was back in the 90s/Early 00s?

If it was sshd, the entire internet would be pretty messed up right now 

I did see some folks suspecting that their admin workstations had been owned, and that keyloggers/data disclosures on Windows & Mac boxes had been used to gain access. Makes some sense when thinking about the recent Java and Flash vulnerabilities...just a reminder to keep your administration tools as secure as your servers, or else it doesn't matter how fancy your security is if an adversary can piggyback.


----------



## throAU (Feb 22, 2013)

SirDice said:
			
		

> Couldn't agree more. Unfortunately there are a lot of people out there that think OS A is better than OS B and they don't need to secure their stuff. Because they are in the *mistaken belief there's something magical called unix* that will prevent any and all infections or hacks.



Exactly, unfortunately.

Sure, you can reduce your exposure somewhat by going for a less popular or more cut down OS, due to the number of vulnerabilities found - but *if a vulnerability is found*, you need to fix it or mitigate it, regardless of OS.


----------



## Crivens (Feb 22, 2013)

Terry_Kennedy said:
			
		

> The big problem is the customized stuff on the three 6.x boxes. Like many businesses, we have some old code. Would it be good to re-write it from scratch? Probably. Does that make any sort of business sense at all? No.



Yes it does.

That business case is called "restart after being owned by evil hackers", and has to be payed in downtime and PR budget. More often than not this case is closely linked to something called "chapter 11", so it does not seem to be a particular profitable case. And after all, it's a case which is not initiated by your company but someone else.


----------



## kleinart (Feb 23, 2013)

throAU said:
			
		

> Exactly, unfortunately.
> 
> Sure, you can reduce your exposure somewhat by going for a less popular or more cut down OS, due to the number of vulnerabilities found - but *if a vulnerability is found*, you need to fix it or mitigate it, regardless of OS.




No one can really argue with any of these comments.  But they're also not really constructive to this thread, IMO.

In my case, I'm fairly sure DNS/bind was my attack vector.  My box was a hobby box, running SSH (locked down, and for my remote shell access/bastion host) and DNS.  Being that it was so limited, I didn't think much of it.  Lesson learned.

Agree, as a business - if you have assets, you need to protect them appropriately.  AND, if you are compromised, the appropriate response is warranted.  And, you should fully rebuild a compromised system (regardless of who you are); I did.  Professionally, yes you need to maintain your stuff.  Hobby systems - one person managing 15+ home systems - easier said than done.

What will NOT happen, as things currently are, will be all systems patched immediately - or even maintained perfectly.  In direct response to the above quote ... there are vulnerabilities found daily, weekly and monthly ... and within a 'short time' you may not have immediate patches available - your dist may not be maintained and you need to do major work to upgrade your system.  Or, in other words, patches probably take too much time and effort as-is, and in the future will probably be more automatic and self-protecting systems and code - just needs evolution time.  (Windows and other desktop OS's - OSX, etc have come far in this regard - automatic updates, etc.)  In this case for me, patches weren't available - I had to upgrade the major revision of the OS... at the time I just figured that was too much effort as I didn't perceive a risk to the box.  I was quite surprised the box got owned - with its limited daemon exposure.

// Wish more effort was put into identifying and classifying root kits, tacking trends, tracking down countries they come from, prosecuting as possible, etc, etc.  Probably hugely controversial - and is currently a profit item ... but non-profit / for the benefit of everyone ... free systems that probe for vulnerabilities and can notify their owners, etc.  Much like I can sign up to receive spam/abuse notices from my IP space, etc.  If hackers can find me, why not whitehats find me first and try to be proactive and help?
I personally already blackhole all of China and N Korea, and will probably add Russia to my list.  Personally, on my hobby net, I have no legitimate reason to really be communicating with those countries, plus I find them to be a risk and not responsible in maintaining law/order/security... let alone the spam issues that had existed (lesser lately than in the last few years).


----------



## Terry_Kennedy (Feb 28, 2013)

SirDice said:
			
		

> It's unclear how exactly they got in but my guess is some broken web application. They then used this 0-day Linux kernel bug to gain root and backdoor sshd(8).


It could be this (more info here).


----------



## SirDice (Feb 28, 2013)

Terry_Kennedy said:
			
		

> It could be this (more info here).



I wasn't too far off when I said it was probably a broken web application


----------



## throAU (Mar 1, 2013)

This is a bit long but...




			
				kleinart said:
			
		

> No one can really argue with any of these comments.  But they're also not really constructive to this thread, IMO.
> 
> In my case, I'm fairly sure DNS/bind was my attack vector.  My box was a hobby box, running SSH (locked down, and for my remote shell access/bastion host) and DNS.  Being that it was so limited, I didn't think much of it.  Lesson learned.



Hobby box - lesson learned.  But given the number of holes, guessing at the attack vector is only a guess.  And it's no real use anyway if you're not going to actually patch the holes  



			
				kleinart said:
			
		

> What will NOT happen, as things currently are, will be all systems patched immediately - or even maintained perfectly.  In direct response to the above quote ... there are vulnerabilities found daily, weekly and monthly ... and within a 'short time' you may not have immediate patches available - your dist may not be maintained and you need to do major work to upgrade your system.



Updating is not a major undertaking if you have a maintenance routine, keep up to date with vulnerabilities and apply updates as and when required.  Do you follow the mailing lists, as recommended for FreeBSD users?  Do you subscribe to the CERT mailing list (I highly recommend this as it will also give you work-arounds if a patch is unavailable or not possible to apply)?

If the answer to that is "no" then maybe you should.

Yes, holes are found daily.  However it is fairly rare that they are exploited in the wild as soon as they are discovered.  Also, throwing your hands up and saying "holes are discovered daily!" is not going to secure your boxes.  You do what you can, and normally this is enough.  If you get owned via a hole that has not been patched (a zero day), then so be it, but at least you tried.  Getting owned via a hole that was patched 3 years ago is just being negligent.

The brakes on your car may not be sufficient to bring you to an emergency stop in every conceivable potential accident scenario either, but this doesn't mean that the car manufacturer just throws their hands in the air, says "these brakes won't stop the car in time every time!" and give up fitting brakes.

It's about risk *management*.  *Eliminating *risk is virtually impossible, but you can significantly reduce the risk to a tiny fraction of what it would be if you were running totally unpatched.




			
				kleinart said:
			
		

> Or, in other words, patches probably take too much time and effort as-is,



*Probably*?  I.e., you don't know?  Security updates generally take perhaps 5-10 minutes to apply.  Schedule a brief outage window, get it done.  It's a lot less work than recovery from an intrusion.

Release upgrades take a while (say, an hour or two?) but this is why you do an upgrade in test first, find out what breaks and then do it in production once you're happy.  However, release support schedules are well known and you have a year or more to work out your migration strategy in advance.

Yes this is work.  It is work required to adequately support ANY platform - be it Windows, Linux, FreeBSD or whatever.



			
				kleinart said:
			
		

> and in the future will probably be more automatic and self-protecting systems and code - just needs evolution time.  (Windows and other desktop OS's - OSX, etc have come far in this regard - automatic updates, etc.)
> 
> In this case for me, patches weren't available - I had to upgrade the major revision of the OS... at the time I just figured that was too much effort as I didn't perceive a risk to the box.  I was quite surprised the box got owned - with its limited daemon exposure.



You're aware of FreeBSD-update?

The fact that you were unaware of the risks leads me to believe that you aren't following the CERT or FreeBSD mailing lists.  I would recommend doing so, so that you are aware of the risks involved, when they are discovered so you can keep on top of things - rather than just leaving the box un-patched until somebody else owns it.




			
				kleinart said:
			
		

> // Wish more effort was put into identifying and classifying root kits, tacking trends, tracking down countries they come from, prosecuting as possible, etc, etc.



Tracking, reporting, classifying risk level, trends, etc. is all already done.  You need to follow the relevant mailing lists.



In summary:

This is a learning phase most of us go through (I've been through it myself back in 1999, having had a Linux box get owned via an old version of sendmail).  

Initially it's like "Yeah, unix is so much more secure than windows!  I'm immune to all these threats!".

After a while, either being owned yourself of seeing others get owned, you realise it's not a magic bullet.  Yes, FreeBSD is more secure than Windows out of the box - but no software is perfect, and thus far throughout the history of computing, it's only a matter of time before virtually all software has holes found in it.

Once those holes are found, they need to be either closed, or mitigated via some other method (filtering access, etc).  If your box is on the internet, you ARE a target.  If you're running unpatched, you will eventually get owned, no question - it's merely a matter of when.


----------

