# System Integrity check for system installed in a datacenter?



## nosystem (Feb 10, 2013)

Hi!

I would like to know if there is any way to check the integrity of a system installed in a datacenter? 

If I didn't install the system myself but it's set up by the guys working in the datacenter, how can I know that it has no rootkit/trojan/spyware/whatever - if possible at all?

I was thinking that it theoretically would be possible to install another system from the same version on another machine and compate both systems, would that do the trick?

Another option would be to ask the system to be installed (for example) with FreeBSD 8.x and then upgrade it manualy from sources to for example 9.1. In that cases, would any files from the older version 'survive' in my 9.x install?

Thank you very much


----------



## graudeejs (Feb 10, 2013)

security/tripwire, security/aide, security/integrit

You may also find this interesting:
http://forums.freebsd.org/showthread.php?t=4108


----------



## swa (Feb 10, 2013)

Hi, 

The freebsd-update(8) utility has a builtin IDS and can detect modified files from the *base* system. See Chapter 25.2.4 from the FreeBSD Handbook  for a way to compare the installed system against a known good copy. 

In the ports collection there are some rootkit scanners but this is not a guarantee that there is no backdoor installed. I think only way to make sure is probably do a clean install yourself. Tools like aide are useless in this case because you need to start with a clean install.


----------



## nosystem (Feb 10, 2013)

Thanks for your answer!

My problem is that I have no way to know if the 'first' system is safe, hence I can't really create a database for integrit - ande & tripwire can be good to check if there is some weird activity, in case the base system that have been installed was corrupted...

Does anyone here knows if a version upgrade would remove all software from the older version (ie 8.x to 9.1) hence destroying any potentially trojaned files?

Thanks again for your answer though, and I'm going to read that topic


----------



## graudeejs (Feb 10, 2013)

nosystem said:
			
		

> Thanks for your answer!
> 
> My problem is that I have no way to know if the 'first' system is safe, hence I can't really create a database for integrit - ande & tripwire can be good to check if there is some weird activity, in case the base system that have been installed was corrupted...
> 
> ...



It would not


----------



## fonz (Feb 10, 2013)

nosystem said:
			
		

> Does anyone here knows if a version upgrade would remove all software from the older version (i[red].[/red]e[red].[/red] 8.x to 9.1) hence destroying any potentially trojaned files?


A version upgrade updates the base system, but leaves ports/packages (as well as additional software compiled and installed manually) alone.


----------



## AlexJ (Feb 10, 2013)

nosystem said:
			
		

> I would like to know if there is any way to check the integrity of a system installed in a datacenter?


1.) Only if it is your hardware(that you can trust) that is on collocation in a datacenter where your box is wrapped around with a huge chain and locked with an unbreakable lock.



			
				nosystem said:
			
		

> If I didn't install the system myself but it's set up by the guys working in the datacenter, how can I know that it has no rootkit/trojan/spyware/whatever - if possible at all?


2.)The whole point of a rootkit - to be a part of operation system, which means it can hook up any API and control/bypass any requests to itself.
That is it. If rootkit is a kernel module it can hide itself, supply checksums that it want, show RAM size as it want... anything... 
If OS got a rootkit, you can't do anything with it. Period.
3.) The one only effective method to analyze operation system, it is boot from trustful CD and inspect filesystem in question with one condition - see paragraph 1.) above.



			
				nosystem said:
			
		

> I was thinking that it theoretically would be possible to install another system from the same version on another machine and compate both systems, would that do the trick?


See 2.)



			
				nosystem said:
			
		

> Another option would be to ask the system to be installed (for example) with FreeBSD 8.x and then upgrade it manually from sources to for example 9.1. In that cases, would any files from the older version 'survive' in my 9.x install?


That is all depend on a quality of a rootkit. If it smart enough, it can control this situation too, so - no, it won't work in a way when you want to be sure 100%  that a system isn't infected.

Back to the paragraph 1.)...
Did you inspect resent hardware for a subject such as "Intel Anti-Theft Service", "McAfee Anti-Theft", "Norton Anti-Theft", "CompuTrace LoJack" by absolute.com?
Hardware already has embedded "phirus"(BIOS virus). BIOS has a official backdoor and CPU has supports for it(anti theft platforms,  CPU with a kill switch).

There is a bunch of viruses that already do BIOS infection and ironically use "Anti-Theft" features and easily survive OS re-installation (first example that Google return: Trojan.Mebromi)

So, back on topic... you can't trust even to a fresh hardware from a store (that by the way infect your purchase with a bunch of trial ad-ware) because hardware isn't trustful anymore.
Looks like a paranoia?  but it isn't, it just a facts that you should know if you care about security/privacy and paranoia


----------



## AlexJ (Feb 10, 2013)

graudeejs said:
			
		

> security/tripwire, security/aide, security/integrit



mtree(8) do the same integrity checking for free(to compare with security/tripwire)


----------



## nosystem (Feb 10, 2013)

> The freebsd-update(8) utility has a builtin IDS and can detect modified files from the base system. See Chapter 25.2.4 from the FreeBSD Handbook for a way to compare the installed system against a known good copy.
> 
> In the ports collection there are some rootkit scanners but this is not a guarantee that there is no backdoor installed. I think only way to make sure is probably do a clean install yourself. Tools like aide are useless in this case because you need to start with a clean install.



Thanks!



> A version upgrade updates the base system, but leaves ports/packages (as well as additional software compiled and installed manually) alone.



Ok, that's great to know, thanks! So _all_ the base system would be upgraded _for sure_, right?



> 1.) Only if it is your hardware(that you can trust) that is on collocation in a datacenter where your box is wrapped around with a huge chain and locked with an unbreakable lock.



Interesting answer. And I'm afraid that I agree with you. You have to trust the datacenter AND the hardware, and I was obviously missing that point...



> 2.)The whole point of a rootkit - to be a part of operation system, which means it can hook up any API and control/bypass any requests to itself.
> That is it. If rootkit is a kernel module it can hide itself, supply checksums that it want, show RAM size as it want... anything...
> If OS got a rootkit, you can't do anything with it. Period.



Wow. That's a point. So basically if there is/can be a _"good"_ rootkit there is probably nothing that can be done, except installing a new system. That totally make sense.

I have to read more about rootkits and maybe start playing around writing one to understand that better. Any recommendations (books, other ressources) about that?




> 3.) The one only effective method to analyze operation system, it is boot from trustful CD and inspect filesystem in question with one condition - see paragraph 1.) above.



Ok - or reinstall a clean system (if possible)...



> That is all depend on a quality of a rootkit. If it smart enough, it can control this situation too, so - no, it won't work in a way when you want to be sure 100% that a system isn't infected.



Ok, you can't be 100% sure but most rootkits wouldn't survive the process, right? I mean even if that solution isn't perfect, it might still be better than nothing I guess...



> Back to the paragraph 1.)...
> Did you inspect resent hardware for a subject such as "Intel Anti-Theft Service", "McAfee Anti-Theft", "Norton Anti-Theft", "CompuTrace LoJack" by absolute.com?
> Hardware already has embedded "phirus"(BIOS virus). BIOS has a official backdoor and CPU has supports for it(anti theft platforms, CPU with a kill switch).
> 
> ...



That's the most interesting part of your answer, because I didn't think about that, and that's pretty obvious. Hardware _isn't safe anymore_. 

So the whole point is trust. Will the datacenter sell me infected software and/or hardware.

In my opinion, I'd believe that if there had to be a problem it would most likely be a software one and as anyway I can't influence on the hardware (rented machine) but have some power over the software (root access) I might do my best to secure that part. _Knowing that it won't be perfectly secure anyway_ - that brings a weird feeling...

So the only way would probably be to get some hardware that I feel like I can trust, inspect it, then setup a clean FreeBSD system myself, go with my own car to the datacenter, put the server in it's rack, and pray that nothing happens. If the whole file-system is encrypted though, it might be harder to setup a software rootkit in it without system access and at least I could have a 'known good' copy of the installed base system that way...

Makes it pretty impossible to make sure that a _rented_ server is trustworthy... :/



> mtree(8) do the same integrity checking for free(to compare with security/tripwire)



Thanks, will check that out. Won't help to know if the system the datacenter did install is clean, but still an interesting tool to check the evolution of the system... once I have to way to be sure that the base one is clean!


----------



## AlexJ (Feb 11, 2013)

nosystem said:
			
		

> Wow. That's a point. So basically if there is/can be a "good" rootkit there is probably nothing that can be done, except installing a new system.


If a rootkit infected BIOS, it will recover itself even after fresh installation from a trustful CD.
Infected code in a BIOS control access to a hard drive and usually drop some shit to a boot record that later using OS API to access some external sites to grab an actual rootkit and infect new, fresh system. (Simplified workflow)

That was the point of Microsoft to require hardware makers to use cryptographically signed "UEFI secure boot" if they want to use Windows 8 with their products.
(While idea of cryptographically signed "UEFI secure boot" is a good one, it may push out of business operation systems that doesn't have such power as Microsoft has and can't supply hashes for "UEFI secure boot" to all motherboard production plants)
There is work that insure originality of non modified operation system by employing UEFI to protect OS on early boot stage(Thanks to Matthew Garrett and others who pick this), but it isn't finished yet.



			
				nosystem said:
			
		

> I have to read more about rootkits and maybe start playing around writing one to understand that better. Any recommendations (books, other ressources) about that?


*"FreeBSD Device Drivers: A Guide for the Intrepid"* ISBN-10: 1593272049  ISBN-13: 978-1593272043

*"Designing BSD Rootkits: An Introduction to Kernel Hacking"* ISBN-10: 1593271425 ISBN-13: 978-1593271428



			
				nosystem said:
			
		

> Ok, you can't be 100% sure but most rootkits wouldn't survive the process, right?


I didn't see a rootkit for FreeBSD that employing cooperation with infected BIOS, but it isn't a fact that it doesn't exist, because such parasites already exist for the Windows and Linux.



			
				nosystem said:
			
		

> So the whole point is trust.


If you don't have access to equipment that allow analyze hardware and crew who understand how to use such equipment, then - yes, inspecting hardware isn't a task for us, regular individuals.



			
				nosystem said:
			
		

> Will the datacenter sell me infected software and/or hardware.


I doubt on that. Datacenters is a pretty competitive market and their managers looking for any possible ways to drop expensive.
So, for a regular datacenter embedding additional software - it is an additional financial loading.
Most problem may come with outsourced personal that may running datacenters and here is more probability that something is behind a scene.
If you are on a collocation with your own box, then IMHO it is a much more trustful solution.



			
				nosystem said:
			
		

> So the only way would probably be to get some hardware that I feel like I can trust, inspect it, then setup a clean FreeBSD system myself, go with my own car to the datacenter, put the server in it's rack, and pray that nothing happens.


I saw once with my own eyes on collocation a box that was literally wrapped with a chain that prevented opening a box  
While it is a "good" over protection  , I think that a simple seal and simple lock on a box may help to stop climb inside a box datacenter's personal and curious collocation neighbors.
Private cage is also a good choice if you are going to keep very sensitive data and can afford its price. 
Most datacenters are SAS 70 certified so it is already a good level of trust.



			
				nosystem said:
			
		

> If the whole file-system is encrypted though, it might be harder to setup a software rootkit


If you will use standard encryption it will require you to supply password on any rebooting which will require you to buy some KVM over IP box to be able to do that.
Anyway I think KVM over IP is "must have" for collocation, just check with datacenter if they allow additional "on top small boxes".
Many datacenters provide free "power strip" (that allow to switch off/on a box in case of complete unresponsive), but some offer it at additional cost or don't provide it at all, so check it too.


----------



## fonz (Feb 11, 2013)

AlexJ said:
			
		

> *"Designing BSD Rootkits: An Introduction to Kernel Hacking"* ISBN-10: 1593271425 ISBN-13: 978-1593271428


Seconded. I have that book and find it very instructive (provided one has a decent working knowledge of C).


----------



## Nukama (Feb 11, 2013)

There is coreboot for supported motherboards, where a virus in the BIOS can be overwritten by a audited BIOS. 

But there are other flash chips in a normal system, where compromise could happen. 
Be it the firmware of your RAID controller, VGA card, NIC or BMC/IMPI...

Note that even an audited coreboot BIOS could be compromised by an successful attack on the OS (if the flash chip is in place, and not removed with a pushpin).


----------

