Chinese manufacturer puts hardware backdoor onto Supermicro server boards.

[coppercone2]

LOL you've never been in a DC have you?

Even the shit ones have better security than the best MoD sites I've been on.

I worked near a high security area. Secured for reasons unknown and theories many. (I heard it was something like Die Hard 4 but I also heard a few other strange theories about it).  I also heard it was armed to the teeth with all sorts of military weapons.

Also keep in mind TEMPEST guidelines are mainly prevent emissions of secure information, not one way signals to trigger a sensitive circuit.. but for re-radiaton I think the installation would need to be other wise compromised by bad technicians or design.
So long you don't show up there at night it was actually pretty tame. At night they would investigate EVERYTHING. Unmarked cars too. Was basically told if I went in front of the building at night I might end up with problems.

During a day you could get close though. Also they did not have provisions as far as I know for weird shit like drones, people on some kind of flying vehicles (big quad etc) or other absurd ideas.

Even a military base will only inspect some % of incoming cars because of time delays. Unless its under lock down.

[mtdoc]

Realistic answers might include: Russia - detracts from the various investigations into their interference into US politics, plus they hate China. Domestic political groups - stir up righteous patriotic fervour with mid-terms coming (against: maybe rather too competent an operation for political rabble rousing). Israel - again, mid-terms, electing right wing pro-israeli candidates might make a little sense but not very much, but the Israelis have demonstrated in the past that they are prepared to do stupidly destructive things to gain a little advantage for themselves so it's not completely beyond reason. Any other sensibly plausible actors?

Any other plausible actors?  Duh - yes - the USA/Western corporate/ MIC/"deep state" which has a decided motive - that is to drum up support for a new cold war with China. For anyone paying attention to the non-tech geopolitical world it is obvious that this is going down. Mike Pence the Vice F'ing President of the USA just gave a speech that basically said as much.   I was just a week ago  (before this story broke) listening to an interview with a Hong Kong based (American) well regarded macroeconomic  guru who pointed out that there is a concerted effort to question the China based supply chain.

Please don't try to tie this in to the anti - Russia hysteria.  I'm surprised to see that so many here have been duped into believing the mainstream media narrative about Russia (and no, there has been no actual evidence presented of their "hacking" the US election - just accusations, allegations, and indictments (another form of allegations) - with no public available evidence - just intelligence agency claims and Robert "Iraq has WMDs" Muellers assertions).

Again - the CIA has a long history of employing operatives in major media outlets to create a narrative and further a political agenda. This is not tin foil hat stuff, but well documented historical fact.

[bd139]

No just the caught red handed murder of British citizens and the massive moderation battles against Russian troll farms on Reddit and mainstream media sites etc (yes that is actually happening because the moment they hellban Russian netblocks it goes DEAD). But this is not an us and them thing. Everyone is universally being a dick by trying to make the other person look like a dick. This is covering up for being a little tiny irritating dick and trying to avoid pissing off the dicks that live in the country. And some people are being really big dicks. Massive dicks. Massive orange dicks. So much so that it ends in a dick waving contest. Lots of dicks. That's it.

Geopolitics isn't really anything to do with this thread though.

Neither is dicks.

[mtdoc]

Massive dicks. Massive orange dicks. So much so that it ends in a dick waving contest. Lots of dicks. That's it.

Geopolitics isn't really anything to do with this thread though.

Neither is dicks.

LOL, yes lots of dicks on the world geopolitical stage.  Perhaps figuratively large ones but literally.....(My hands are not small!). 

In any case, I would argue that it has everything to do with geopolitics, corporate power, money and the integrity of the current supply chain.

[bd139]

Citation for my last comment for reference. Someone had enough of their shit: https://themoscowtimes.com/news/infamous-st-petersburg-troll-farm-set-on-fire-63130

Massive dicks. Massive orange dicks. So much so that it ends in a dick waving contest. Lots of dicks. That's it.

Geopolitics isn't really anything to do with this thread though.

Neither is dicks.

LOL, yes lots of dicks on the world geopolitical stage.  Perhaps figuratively large ones but literally.....(My hands are not small!). 

In any case, I would argue that it has everything to do with geopolitics, corporate power, money and the integrity of the current supply chain.

I think we should start with credibility. If it turns into geopolitics once we've established credibility then fine. But we haven't established credibility. There is one source and the source has been figuratively kicked in the face repeatedly over the last few days because they are silent on it and have put forward no sources. Even self-proclaimed sources said they got it wrong and extrapolated.

So probability and credibility before blame.

[mnementh]

Credibility is simple: China is where they make ALL our shit. A factory there SOMEWHERE is not only the likeliest vector for such a broad-based attack, it is by far the only sensible vector.

It is not a question of IF... it is a question of WHEN this or something functionally equivalent will happen.

Most of the players involved have a long history of diversionary tactics; it is not beyond any of them to deliberately set this entire scenario in motion to either distract from, or to desensitize the public in preparation for the inevitable public discovery of a known similar threat.

This "outrage after outrage" sensory bludgeoning tactic is how in just a few short decades we went from being a semi-civilized nation as depicted in The West Wing to real-life Idiocracy with spam-sucking trailer trash in the White House leading the nation down the road to Nuclear War and all you ever hear about is the lowest 10% cheering him on. 

mnem
The road to Hell is paved with sociopathic Cheetos. And oil.

[mtdoc]

Citation for my last comment for reference. Someone had enough of their shit: https://themoscowtimes.com/news/infamous-st-petersburg-troll-farm-set-on-fire-63130

No argument that troll farms exist. USA intelligent agencies have their own as well. Countries efforts to influence other countries populaces opinions is an age old tradition. Before the internet it was the "voice of America" and foreign equivalents, before radio it was solely via print media.

But troll farms are a much different than the allegations and Russia hysteria claims that have been repeated ad nauseam in the US media since the 2016 election.  Too many in this country seem psychologically unwilling to look at the internal issues that gave us our orange idiot in chief and are quick to buy convenient blame on "outsiders"  Ah, such is  human nature I suppose.

I think we should start with credibility. If it turns into geopolitics once we've established credibility then fine. But we haven't established credibility. There is one source and the source has been figuratively kicked in the face repeatedly over the last few days because they are silent on it and have put forward no sources. Even self-proclaimed sources said they got it wrong and extrapolated.

So probability and credibility before blame.

I agree. Credibility is key.  It seems to me as a non-expert electronics hobbyist, that if this hardware hack exists, providing physical evidence for evaluation by neutral 3rd parties should not be hard. If no such hardware is forthcoming, the lack of credibility will be confirmed.

[TimNJ]

https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom

I dunno still sounds like "We heard from someone, somewhere, that something happened."

[bd139]

Yep. Shit or get off the pot.

I want to see evidence and analysis published.

Also I have never once heard of Yossi Appleboum or CyberSeal and I'm mostly in that sector.

[bd139]

Interesting picture posted elsewhere...

[Mr. Scram]

Interesting picture posted elsewhere...

One problem with the story seems to be that many people don't seem to understand what is actually possible. Even many people here, and that this was possible years ago now.

All without saying this story actually checks out.

[bd139]

It's possible but unlikely which is the thing. Look at the unit cost of that implant for the NSA. There's an Aspeed SoC on the server boards with an ARM core. Why the hell not just go for the firmware for that? Perhaps that is what happened and Bloomberg are just dumbasses (likey as the reporting is terrible so far).

There's just no logic in any of this.

[mtdoc]

https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom

I dunno still sounds like "We heard from someone, somewhere, that something happened."

Yes. From the linked article:

"Appleboum said his concern is that there are countless points in the supply chain in China where manipulations can be introduced, and deducing them can in many cases be impossible. “That's the problem with the Chinese supply chain,” he said."

Bingo.

Yep. Shit or get off the pot.

I want to see evidence and analysis published.

And it needs to be more than just a one off example. If this is real, there will be multiple examples of the suspect chip being found in the wild.   After all, I would not put it past the intelligence agencies supplying one of their own chips (such as the one you linked) and having it held up as evidence.

[Mr. Scram]

It's possible but unlikely which is the thing. Look at the unit cost of that implant for the NSA. There's an Aspeed SoC on the server boards with an ARM core. Why the hell not just go for the firmware for that? Perhaps that is what happened and Bloomberg are just dumbasses (likey as the reporting is terrible so far).

There's just no logic in any of this.

I'm not disagreeing with that, I'm just saying that people reach what's possible the right conclusion for the wrong reasons. They dismiss the possibilty because they don't consider it technologically viable. We have fairly convincing proof similar technology exists and may very well have been deployed. It's just questionable whether that happened here.

Even after having a good portion of the technological portfoilio of the NSA publicly available, people still gravely underestimate the capabilities of these organisations.

[mtdoc]

It's worth noting that the NSA chip in bd's post is much, much larger  than the "grain of rice" sized chip claimed in the Bloomberg article.  Of course that published NSA chip data is several years old now - so no doubt similar tech could be smaller now - but "grain of rice sized"?. Dunno.

[Mr. Scram]

It's worth noting that the NSA chip in bd's post is much, much larger  than the "grain of rice" sized chip claimed in the Bloomberg article.  Of course that published NSA chip data is several years old now - so no doubt similar tech could be smaller now - but "grain of rice sized"?. Dunno.

You could argue about that. As you say, the information we have is dated at this point and the budget is ridiculous. But I too simply don't know.

[mtdoc]

It's worth noting that the NSA chip in bd's post is much, much larger  than the "grain of rice" sized chip claimed in the Bloomberg article.  Of course that published NSA chip data is several years old now - so no doubt similar tech could be smaller now - but "grain of rice sized"?. Dunno.

You could argue about that. As you say, the information we have is dated at this point and the budget is ridiculous. But I too simply don't know.

And then we just come back to the point that even if it is possible, is that the smartest  way to achieve the goal?  Why such an easily detectable and traceable tactic? If you're China, why jeopardize the technology supply chain that is the keystone to your economic power?

[Mr. Scram]

And then we just come back to the point that even if it is possible, is that the smartest  way to achieve the goal?  Why such an easily detectable and traceable tactic? If you're China, why jeopardize the technology supply chain that is the keystone to your economic power?

If the technology is real, we still don't know whether it's actually China.

[floobydust]

Today another vague update from Bloomberg: https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom

"... A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer Inc. in its network and removed it in August {2018}"
No mention of the actual telecom carrier, other than AT&T and Verizon saying it's not a problem for them.

"... manipulated Ethernet connector" which is metal instead of plastic, for heatsinking the chip inside.

This is saying the H/W trojan is at the Ethernet PHY ?

[tooki]

One problem with the story seems to be that many people don't seem to understand what is actually possible. Even many people here, and that this was possible years ago now.

All without saying this story actually checks out.

I don’t think anyone has claimed the technology isn’t available. What is not credible is the many layers of corporate bureaucracy that would have to be penetrated to alter so many corporate divisions simultaneously. For crying out loud, change management is hard in the best of times. Infiltrating that so that you can change the schematic, the PCB, the testing jigs and test routines, and the validation processes back at the home office in USA for the production samples that are pulled for spot testing? That simply does not sound possible to pull off. Such changes are hard enough when they’re legitimate; doing them covertly just defied credibility.

[ajb]

Thinking about it some more, there are actually some solid benefits to using a hardware implant rather than compromising the firmware or, say, a flash IC. 

It's conceivable that there's some exploit that requires a fairly minimal modification of the firmware binary, and that the location of that modification is easily recognized within the binary by its surroundings.  As long as that specific area of the binary was not changed (which could be unlikely unless that specific area of the codebase was changed), then the malicious device could be capable of compromising any new firmware version, even if the targeted area appears somewhere else within the binary.  Not unlike the infamous Ken Thompson hack.  Even pulling the flash from the board and dumping it externally wouldn't reveal anything amiss.  You'd have to directly sniff the traffic between the embedded controller and the interloper to capture the change to the binary, and even then it's conceivable that the interloper has some sort of context awareness to help avoid detection (not unlike the VW firmware that could detect emissions testing).

Also, somewhat ironically, the fact that almost everyone here is saying that it makes so much more sense to compromise the firmware or one of the existing ICs on the board is something of an argument for NOT doing it either of those ways--after all, it's exactly what anyone would expect!  It would be far sneakier to make a fake passive component that pwns the board because that's such a ridiculous idea that no one would ever bother to do that sort of thing, right?  Just like no one would try to cram a network traffic siphon with a built-in RF transceiver inside of a network jack. . . .

I also think a lot of people are overestimating how easily an extra component or two would be detected.  I mean, I sure as hell wouldn't notice an extra couple of passives on one of my boards between finished a design and receiving the assembled thing, and my boards aren't nearly as complex as a server motherboard.  Plus you would have teams of people working on those things, and no one person is going to know the entire board like the back of their hand.  They're only going to start comparing the finished board to the assembly drawings if something doesn't work, and even then the discrepancy won't be caught if that work is happening at the contractor that installed the malicious parts in the first place.  It all depends on how much of the work Super Micro is farming out, but I imagine that they have their design process down to such a science that it's very rare they have to do component-level debugging.

Of course without more information it's impossible to tell if these benefits likely outweighed the difficulty of implementing the exploit in the way that Bloomberg describes, but still, it's all plausible through a certain lens, which is what makes it so intriguing.

[Mr. Scram]

I don’t think anyone has claimed the technology isn’t available. What is not credible is the many layers of corporate bureaucracy that would have to be penetrated to alter so many corporate divisions simultaneously. For crying out loud, change management is hard in the best of times. Infiltrating that so that you can change the schematic, the PCB, the testing jigs and test routines, and the validation processes back at the home office in USA for the production samples that are pulled for spot testing? That simply does not sound possible to pull off. Such changes are hard enough when they’re legitimate; doing them covertly just defied credibility.

Many people seem incredulous for technological reasons, at least that's my impression. They may be right, but for the wrong reasons.

[ajb]

I don’t think anyone has claimed the technology isn’t available. What is not credible is the many layers of corporate bureaucracy that would have to be penetrated to alter so many corporate divisions simultaneously. For crying out loud, change management is hard in the best of times. Infiltrating that so that you can change the schematic, the PCB, the testing jigs and test routines, and the validation processes back at the home office in USA for the production samples that are pulled for spot testing? That simply does not sound possible to pull off. Such changes are hard enough when they’re legitimate; doing them covertly just defied credibility.

The article claims that this happened on boards that SuperMicro contract out, so that means you only have to compromise that narrow bottleneck where the two companies communicate.  Say you compromise Super Micro's account manager at the subcontractor: He passes you SuperMicro's design package, you tweak it, and send it back, and he passes it on to engineering for validation, DFM review, and eventual production as if it came directly from his customer--and in fact, he very likely has an email from his contact at Super Micro saying "sorry, that design package wasn't the latest revision, please use this new one instead", because surely anyone who would commission such an exploit knows how to spoof emails.  Easy peasy.  In fact, since this purportedly happened at subcontractors to subcontractors, you have a further level of insulation, and excuse for delays in communication and misunderstandings that give you some leeway to operate. 

[technix]

Thinking about it some more, there are actually some solid benefits to using a hardware implant rather than compromising the firmware or, say, a flash IC. 

It's conceivable that there's some exploit that requires a fairly minimal modification of the firmware binary, and that the location of that modification is easily recognized within the binary by its surroundings.  As long as that specific area of the binary was not changed (which could be unlikely unless that specific area of the codebase was changed), then the malicious device could be capable of compromising any new firmware version, even if the targeted area appears somewhere else within the binary.  Not unlike the infamous Ken Thompson hack.  Even pulling the flash from the board and dumping it externally wouldn't reveal anything amiss.  You'd have to directly sniff the traffic between the embedded controller and the interloper to capture the change to the binary, and even then it's conceivable that the interloper has some sort of context awareness to help avoid detection (not unlike the VW firmware that could detect emissions testing).

Also, somewhat ironically, the fact that almost everyone here is saying that it makes so much more sense to compromise the firmware or one of the existing ICs on the board is something of an argument for NOT doing it either of those ways--after all, it's exactly what anyone would expect!  It would be far sneakier to make a fake passive component that pwns the board because that's such a ridiculous idea that no one would ever bother to do that sort of thing, right?  Just like no one would try to cram a network traffic siphon with a built-in RF transceiver inside of a network jack. . . .

I also think a lot of people are overestimating how easily an extra component or two would be detected.  I mean, I sure as hell wouldn't notice an extra couple of passives on one of my boards between finished a design and receiving the assembled thing, and my boards aren't nearly as complex as a server motherboard.  Plus you would have teams of people working on those things, and no one person is going to know the entire board like the back of their hand.  They're only going to start comparing the finished board to the assembly drawings if something doesn't work, and even then the discrepancy won't be caught if that work is happening at the contractor that installed the malicious parts in the first place.  It all depends on how much of the work Super Micro is farming out, but I imagine that they have their design process down to such a science that it's very rare they have to do component-level debugging.

Of course without more information it's impossible to tell if these benefits likely outweighed the difficulty of implementing the exploit in the way that Bloomberg describes, but still, it's all plausible through a certain lens, which is what makes it so intriguing.

IMO I doubt the practicality of cramming that much processing power and RF frontend in a chip of that size.

I have a different theory then: the compromised chip is still the ASPEED one for its processing power, convenient access to host memory (over PCIe) and networking (over IPMI, or heaven forbid a network port shared with the host server since ASPEED actually support that as advertised,) but there is indeed a fake passive involved. The fake passive would be connected to the ASPEED chip using one of its serial debug pins to inject code. For a uncompromised board the ASPEED chip is properly set in production mode with, say, a TEST pin tied to ground. For a compromised board a resistor is omitted (or the solder pad being removed or covered before soldering - you get the gist, just leave it floating) so the ASPEED chip boot with production test enabled, and that fake component happen to sit on a serial debug line that can inject code. If you take the ASPEED chip off the board to inspect it would function as normal, the firmware would reveal nothing, and installing a new ASPEED chip does not result in the attack vector go away. This requires just one ASPEED engineer leaking the specs of the production testing pins, some reverse engineer on the public firmware binary, and one manufacturing employee subverted.

[coppercone2]

a fast TDR should go fucking bonkers on a cable implant.