# How China Used a Tiny Chip to Infiltrate U.S. Companies



## drhowarddrfine (Oct 4, 2018)

Perhaps of significant interest to those who use SuperMicro servers specifically:

The Big Hack



> During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.


----------



## ralphbsz (Oct 4, 2018)

This problem is likely not specific to SuperMicro; server motherboards from other vendors are likely neither better nor worse in this respect.  And I would assume that other agencies (such as the non-existing ones that are headquartered in Maryland) have done similar things.  Perhaps not to server motherboards, perhaps to networking hardware and software.


----------



## shepper (Oct 4, 2018)

The chip is described as being the size of a rice grain.  I wonder if a more devious exploit would embed the functionality into an existing chip.

There also has to be a way for the chip to either declare it's presence or respond to a probe.  It should be possible, after analyzing the extraneous chip, to write something to probe existing motherboards.

Another aside, anyone know of any motherboard manufacturers based in the US? Dell? Would be a good stock to buy.


----------



## ralphbsz (Oct 4, 2018)

Dell probably uses contract manufacturers to actually build its hardware, as do most other companies (HP, Lenovo, ...).  Most PC board contract manufacturing happens in China these days.  I only know one contract manufacturer that is not heavily China based, and that is Jabil Circuits, but I have no idea for whom Jabil builds boards, nor how big their market share in computer manufacturing is.

Why do you think this would make Dell a good investment?  For a very long time, computer users (big and small) have not cared about security, whether it is CPU vulnerabilities (Intel, AMD, ...), motherboard service processor vulnerabilities, the NSA back door in encryption and Windows, and so on.  For a long time computer customer have voted with their feet to buy the best value for their application, and I don't expect that to change.  Whether Dell is a good investment or not depends on many things; the biggest factor being how they'll deal with the fact that the market for servers and server infrastructure (storage, networking, cooling) is going to be eaten by cloud providers.  Or how to get consumers cost-effective and high-quality laptops (serving both the desired of the people who buy a $200 laptop at Costco and who buy a $2000 solid-metal laptop for corporate use).  A security issue such as this is minor compared to other drivers of Dell's stock price.


----------



## shepper (Oct 4, 2018)

ralphbsz said:


> Why do you think this would make Dell a good investment?



I was asking the question.



shepper said:


> Dell?





ralphbsz said:


> .... the market for servers and server infrastructure (storage, networking, cooling) is going to be eaten by cloud providers.



I was under the impression that cloud providers were just remote servers.


----------



## ralphbsz (Oct 4, 2018)

Yes, but the big cloud providers (Amazon, Google, Microsoft, ...) or the big computer users (Facebook, Apple, ...) do not buy servers or infrastructure in the normal market; they get nearly everything custom built.  I don't think you could find a Dell server in an AWS or Google data center.


----------



## recluce (Oct 4, 2018)

My old employer in Germany did good business selling Swedish firewall technology (at least in the mid-2000s), because companies and government agencies were concerned about backdoors in US products. The only drawback: the system boards came from China....


----------



## Deleted member 30996 (Oct 4, 2018)

I am not the least surprised, more surprised we were naive enough it happen. I thought they already got caught backdooring routers en route to the US.


----------



## ralphbsz (Oct 5, 2018)

No, modifying hardware in transit is what the NSA does.  You think I'm kidding?


----------



## Deleted member 30996 (Oct 5, 2018)

ralphbsz said:


> You think I'm kidding?



Not at all. I don't put anything past them and you always know what you're talking about. They did discover a possible  backdoor in routers and VoIP products but I thought they had been doing it en route.

I don't know why any of this should come as a shock. Cyberwarfare is nothing new and you have to give credit where credit is due for ingenuity. It's part of the bigger picture as I see it. The expansion of territory into the China Sea, theft of intellectual property rights, upgrading their military, loaning us more money that should possibly exist.

And then there's Russia. Forget taking down the powergrid. Their contingency plans for being on the losing end of a war with the US are nuking the Yellowstone Supervolcano and detonating Superweapons underwater off the coast to cause tidal waves who knows how tall. Wonder who thought of nuking Yellowstone? That is evil genius.

I don't have a very optimistic outlook for the years ahead.


----------



## shepper (Oct 5, 2018)

ralphbsz said:


> Yes, but the big cloud providers (Amazon, Google, Microsoft, ...) or the big computer users (Facebook, Apple, ...) do not buy servers or infrastructure in the normal market; they get nearly everything custom built. I don't think you could find a Dell server in an AWS or Google data center.



Even though I live in WA, with server farms running clustered around the hydroelectric powerplants and Columbia River wind forms, I do not know for sure what the Cloud providers use for hardware or where they have it manufactured.

I have been following AMD's quartelrly financial reports and they are making significant inroads with the EYPC.
This Forbes article says many of the larger Cloud providers have been their customers.

Forbes, AMD/EYPC



> AMD’s strategy for re-entering the server market was straight-forward: pursue the cloud as enterprise lags in adoption. This makes sense. Cloud providers are beholden to no server or CPU company, and they buy in volume. Conversely, enterprise IT organizations, some stung by AMD’s exit from the server market will need a little more convincing that it’s a good idea to invest in EPYC and again in AMD.
> 
> This strategy appears to be working out as AMD has been deployed in cloud giants Microsoft Azure, Baidu, and Tencent, in addition to Yahoo! Japan, Hivelocity and Packet (a bare metal cloud provider). The key here is the “d-word” – deployed.  I have seen many non-Intel server CPUs tested, but very few deployed in volume and this makes a huge difference.
> 
> EPYC’s success in the cloud undoubtedly fueled the doubling of sales (QoQ) and double-digit revenue growth in 1Q of 2018. While recognizing that growth is being compared against very low revenue, the trends are positive.


----------



## ralphbsz (Oct 5, 2018)

The stuff about AMD and server farms is definitely correct.

For several years, there have been several really big customers of computers: The federal government, Amazon, Google, Microsoft, and so on.  The big cloud providers build their own hardware (motherboards, enclosures, networking), but they buy components such as processor chips, disk drives, RAM DIMMs, and smaller stuff.  The federal government for the most part buys off-the-shelf servers (but mostly high-end servers, with a lot of that being supercomputers from Cray and IBM); I don't know what hardware they use for their internal secret cloud-like deployments (the big NSA data center in Utah for example, which does not contain supercomputers, and I have never heard what hardware they bought).  Most of that stuff uses Intel CPUs (and obviously a mix of Seagate and WD drives, there being de-facto no other disk manufacturers any more).

Now, the fact that Intel had a de-facto monopoly on CPUs really bugged these big users, in particular the federal government and Google.  So they have been trying to keep the competition to Intel artificially alive: Using IBM Power chips and AMD Opterons for supercomputers, trying to have a little bit of ARM servers and AMD server chips.  And it seems that this has succeeded to the point where now ARM and AMD are beginning to be viable competitors to Intel.  We might be in a situation where in a year or two AMD might have a double-digit percent market share (could be as much as 50%) in the "server" market, which today mostly means big cloud users.  Personally, I would welcome that; the Intel monopoly has made me just as uncomfortable as everyone else.  But what we need to remember: If it weren't for the generosity of "donors" like the national labs and Google, ARM servers and AMD would be dead today.


----------



## DriverBuilder (Oct 5, 2018)

In this article they mentioned "the microchip altered the operating system’s core so it could accept modifications". I wonder is this affects FreeBSD OS?


----------



## Crivens (Oct 5, 2018)

Japanese proverb: "Business is the long sword". And espionage on that level is just business, nothing personal. The problem is that most nations no longer have the ability to build systems from scratch, as a report about the iPhone once documented. What did everybody expect? 
When Boing won the bid to supply the equivalent of AF1 to China (they seem to have had a hidden finance source for being so cheap) some of their personel ended up in jail there because the bird was immediately checked for 'additional equipment' which was also found in the bed posts f.e.

One should check the silicon delivered from anywhere for add ons. One good way to slip something into an ARM core is to make an add of #0 to the PC set the supervisor mode status flag. So you would need to check all these no-op and illegal-op combos if they do as expected.


----------



## Crivens (Oct 5, 2018)

Also of interest.


----------



## tingo (Oct 5, 2018)

DriverBuilder said:


> In this article they mentioned "the microchip altered the operating system’s core so it could accept modifications". I wonder is this affects FreeBSD OS?


Think BMC = IME (Intel Management Engine) and that's where this chip was attached (according to reports), then decide for yourself if it could affect any operating system on that hardware.


----------



## Beastie7 (Oct 5, 2018)

My question is, what can the FreeBSD community do to protect its users (and the platform) against these adversaries and flaws?

Does the community care enough about the issue to advocate for and encourage alternatives? Or do we just want to bicker and regurgitate what some of us already know?


----------



## Deleted member 30996 (Oct 5, 2018)

tingo said:


> Think BMC = IME (Intel Management Engine) and that's where this chip was attached (according to reports), then decide for yourself if it could affect any operating system on that hardware.



I'm much more concerned with the IME than with Intel chip exploits. I can mitigate those to an extent by not allowing scripting in my browser, they worry me as much as a JavaScript trojan.

The only method I've seen described to deal with IME backdoors carries the caveat "Use at your own risk; the methods to disable Intel ME were described as “risky and may damage or destroy your computer.”

I care very much about computer security and learned the importance of it early on, but at a point become somewhat desensitized to what is a never-ending list of things to worry about. I can only maintain a state of hyper-vigilance for about 2 years till what I was so worried about initially slowly becomes a normal part of everyday life and deal with new entries on the list as needed.

That's not to say I become lax or complacent.


----------



## kpedersen (Oct 5, 2018)

ralphbsz said:


> If it weren't for the generosity of "donors" like the national labs and Google, ARM servers and AMD would be dead today.


I always find it quite interesting that the reason why AMD is around in the first place is because IBM at the time refused to be tied down to a single vendor (Intel) for its hardware so it somehow got Intel to license out the rights to its chip designs so that IBM would go with them. This again felt like an artificial deal.

Would this ever happen again? Is Intel too big and would decline the contract terms?

Is Intel quite pissed off that this happened and that it created (I guess it's main competitor) its own competition? I am assuming there is much regret.

Would Intel ever be allowed to buy out AMD? The fact that AMD is originally dependent on Intel's patents etc. surely gives it quite a good argument / case against potential anti-monopoly lawsuits


----------



## shepper (Oct 5, 2018)

Intels' attempt to maintain a monoply was a sustained effort at least up until 2009.  Their tactics went well beyond licensing technology.

AMD litigation with INTEL


----------



## Deleted member 55181 (Oct 5, 2018)

Long story.

https://translate.google.com/transl...-amazon-sprzet-supermicro/&edit-text=&act=url


----------



## bookwormep (Oct 6, 2018)

Some reading material: "Dark Territory: The Secret History of Cyber War," by Fred Kaplan. The author cites some declassified events worth reading.


----------



## ronaldlees (Oct 7, 2018)

For nation state takeover, they won't use trojan signal conditioners on motherboards.  They'll use photonics.


----------



## ikbendeman (Oct 30, 2018)

I miss these tin foil hat posts. It's true that most nations put some kind of backdoor. I was just on votesmart and saw that, here, there has been legislation put forward to install killswitches in cellphones (I guess the lawmakers don't realize those are already there).

The crazy part about this story is how widespread it was before our domestic agencies detected/released knowledge about it.

Most of these agencies put resources towards these things in accordance with market share, which tends to be linked to usability. If you don't want to worry about this kind of stuff, good luck building an invulnerable system. I'll take our US backdoors over the chinese, though, as the chinese tend to leave them more or less open for other actors.

Taking tinfoil hat off now...


----------



## ikbendeman (Oct 30, 2018)

DriverBuilder said:


> In this article they mentioned "the microchip altered the operating system’s core so it could accept modifications". I wonder is this affects FreeBSD OS?



I think this was a CPU level exploit, so technically, it could exploit an OS not yet developed for aforementioned platform.


----------



## Crivens (Oct 30, 2018)

As it turned out, the story was a load of bollocks. Bloomberg jornos are said to get a bonus when a story rocks some share values, and I think SuperMicro is now sending some sueballs their way.


----------



## Phishfry (Oct 30, 2018)

shepper said:


> Another aside, anyone know of any motherboard manufacturers based in the US?


Jabil Manufacturing in the Tampa area.
They build specialty computer products for the government  purchased via a General Dynamics division.
There may be others I am unaware of.



Crivens said:


> As it turned out, the story was a load of bollocks. Bloomberg jornos are said to get a bonus when a story rocks some share values, and I think SuperMicro is now sending some sueballs their way.


Supermicro stock was decimated before this story even hit.
A US designer using communist manufacturers. What could go wrong?


----------



## Phishfry (Oct 30, 2018)

After Snowden and NSA TAO details emerged. All you have to ask yourself.
Do you really think other countries aren't doing this too?
Big companies lying. Say it ain't so.
How many years has TAO existed doctoring up Cisco and Juniper gear?
If your waiting for the government to admit this actually happened then you have a long wait.

Bloomy is fake news but these hijinks aren't exactly groundbreaking.
Embedding a chip in a PCB is totally achievable. Maybe not in mass production but for Tailored Operations.


----------



## Phishfry (Oct 30, 2018)

I can't imagine why China would be interested in Amazon....
https://aws.amazon.com/blogs/publicsector/announcing-the-new-aws-secret-region/


----------



## ikbendeman (Oct 30, 2018)

Name a country that isn't doing it, and you'll name a regime that's no longer in power.


----------



## ralphbsz (Oct 30, 2018)

Phishfry said:


> Jabil Manufacturing in the Tampa area.
> They build specialty computer products for the government  purchased via a General Dynamics division.


They also have assembly plants in Silicon Valley.  I do not know who all their customers are, but I know that some US-based computer companies that are large suppliers to federal agencies use Jabil.  I am not sure that Jabil is purely a US-based company though; they might very well outsource some of the work to low-wage countries.



> Supermicro stock was decimated before this story even hit.
> A US designer using communist manufacturers. What could go wrong?


I've been a small-time Supermicro customer for a long time (mid- or late 90s, I think).  They build good stuff, nicely engineered to be efficient and feature-rich.  Their quality control is a mixed bag; while the hardware quality is pretty good (sheet metal has edges deburred so you don't cut yourself, holes in PC boards are accurate enough so mounting points line up correctly, screws and mounting points are in sensible places), they are not consistent, and things like firmware versions or sudden changes in specifications can be very annoying.  They're not Sun or HP, but also not no-name brand motherboard house.

But referring to them as a "US designer" is a bit wrong.  The company consists mostly of Taiwanese people, has strong business ties to Taiwan, and also employs a significant number of people originally from mainland China.  When working with them, you quickly discover that only a small fraction of their staff speak English.  But working with them has always been pleasant (they're friendly, competent, and responsive, even if there is a layer of translation involved), and their products have generally been good for me.



ikbendeman said:


> Name a country that isn't doing it, and you'll name a regime that's no longer in power.


Exactly.  The only governments that don't use espionage and insurgency are those that are too incompetent, or too poor to afford it.  As an example, look at the Grand Duchy of Fenwick: they had neither good intelligence nor a functioning military, and yet ...


----------



## Phishfry (Oct 30, 2018)

I have owned SuperMicro boards for 20 years now. I have two new X10 motherboards in front of me. On the boxes is printed Designed in the USA. Going into storage I pulled some boxes and all those also say Designed in the USA.

Also their World Headquarters is in San Jose California and thier campus makes up over 7 Office buildings looking at the map.
http://www.supermicro.com/about/contact/
Now I have never personally verified that the boards are designed here. I have to take their word for it.

I am not trying to attack SM. I love them. I could not afford a Dell or HP server.
Personally I am not sure that they were involved with these hijinks. Maybe they design custom boards for big cloud clients like Google and Apple. It would not be out of the realm of possibilities. Maybe they handle the contract manufacturing too.

In my opinion Amazon is being attacked by every intelligence agency as it is now a military target because of the top secret cloud.


----------



## drhowarddrfine (Oct 30, 2018)

Phishfry said:


> On the boxes is printed Designed in the USA.


Designed doesn't mean manufactured and  it would be easy to take those CAD plans and rewire a part in without anyone knowing.


----------



## ikbendeman (Oct 30, 2018)

If you read the story, it said that the chinese plant was contacted by the CCCP or the PLA, who then "requested" that the design be modified to retrofit the backdoor. Whether this story is false or not is somewhat irrelevant unless you own aforementioned hardware or stock, as I can assure you this is most certainly in practice in China, not to mention elsewhere... It's just easier to find it coming from China, as they hold a decent market share for manufacturing. I don't know the last time I bought a PCB that was marked "made in Russia" for example, but that's because the answer could well be never. Also, if the company is an international corporation there are certain situations where it could be mostly, or partially designed in Tiawan or the design could be "implimented" there, but legally labeled "Designed in the USA"


----------



## ikbendeman (Oct 30, 2018)

drhowarddrfine said:


> Designed doesn't mean manufactured and  it would be easy to take those CAD plans and rewire a part in without anyone knowing.



Especially if your state has one of the largest military cyber warfare/intelligence operations in the world and has access to much of the world's manufacturing of said sector. What am I talking about? We all know the CCCP has too much integrity to lie.


----------



## ralphbsz (Oct 30, 2018)

Phishfry: I completely agree, their headquarters campus is very much on the northeastern edge of San Jose.  I drive by there at least 4 times per week, and I've visited before.  If their boxes say "designed in the US", I'm inclined to believe them, they wouldn't lie about stuff like that.


----------

