Impervious Implants & Splintery Supply-Chains

Update 10:21am PST 5Oct2018 (added quite a few links/references to articles mentioned). 
Update 17:32am PST 5Oct2018 (added references to Eclypsium's work)

Every few years a huge "Supply Chain" security news story hits the mainstream that jolts everyone into a state of unease and paranoia. These stories serve as jarring reminders that amidst the dull hum of discontent that we all have about privacy (in our increasingly app-reliant connected-lives), we also have to worry about the hardware vessels that are home all this presumably sketchy software.

If you haven't heard the news, Bloomberg broke a story recently about hardware "spy implants" found in servers made by a company called SuperMicro. These servers were allegedly used by everyone from Amazon to the CIA and Department of Defense. (Amazon released a response statement to this story, which is also a recommended read).

Given that news, we wanted to take a break from our usual postings to share a bit from our perspective. If you dont know who we are and why we may be commenting, please read the next paragraph. If you already do know, skip the next paragraph ;-).

Note: This article will get increasingly more technical as it progresses.

Who we are, and why we are commenting...

We make software for securing networked embedded devices (aka anything that talks on a network). But we also teach the world's most popular hardware/firmware exploitation course, along with some other industry "go to" related trainings (since 2011). Prior to this, we ran a services firm called Xipiter that helped major vendors (cellphone manufacturers, ATMs/Point Of Sale, Gaming Systems, Set-top boxes, governmental agencies, etc) secure their hardware and the software that lives inside. We also made hardware to facilitate this kind of work. In short, we've likely worked on hardware that is in your pocket or that you've recently swiped your credit card through. Our current team is made of veteran security folks that hail from research institutes like MIT, Draper Labs, McAfee, Department of Defense, etc. We've even briefed the U.S. Intelligence Community on hardware security supply-chain issues.

What is an implant?

There many different types of implants. If we use the analogy of hiding a physical "spying" device in someone's home...the possibilities are endless. The device can be configured to do any number of things like:

• record video
• record audio
• allow access to the home when the inhabitant is away
• join their wifi
• spy on their wifi
• monitor their power usage
• detect motion
• send data recordings of the above off via cellular data

The implant in can be whatever the attacker needs it to be to achieve whatever their goal may be. Hardware implants are no different. So all implants (whether hardware or software) are constrained by:

Hardware Implant constraints:

1. What the attacker wants to accomplish.
2. The attacker's technical competence/budget.
3. The stealthiness of the implant (e.g. You would likely notice a whole camera crew hiding in your closet.)
4. The feasibility of the implant to survive change. (e.g. If the implant needed to be installed before you moved in, but only worked well when your furniture was layed out in on specific configuration...like no artwork hanging in front of pinhole cameras!)
5. The "operational cost" of the implant (e.g. if the implant required someone to sneak into your home to gather its data every 24 hours)

How hardware implants tend to work...

Hardware implants tend to be more "operationally expensive" the less and less they make use of software components. What we mean by this is akin to number 5 in the list above.  If a hardware implant is implemented in "pure hardware"  (leveraging little to no "software") the more complex it tends to be for that device to be generally useful to an attacker.

​For example: if the implant only transmits its salacious spy data over close-proximity radio or when an attacker plugging directly into the implant, this requires the attacker (or a co-conspirator) to be in physical contact with the target device. For every thing you would want to spy on, you have to send someone there (paying their travel and accommodations, and risking getting caught) to collect the "spy data". 

This is why "hardware hacking" that only focuses on the hardware is more-or-less "arts & crafts" or an exercise in academic self-importance.

And when weighing those options, if someone is going to be in physical contact with the target, why not just stage a "smash and grab" or do something else "physical" while you are there? Why develop a custom piece of hardware if you're going to need to have someone there to make use of it? This is why "hardware hacking" that only focuses on the hardware is more-or-less "arts & crafts" or acts of academic self-importance. Most effective "hardware hacks" require some element of software to be generally useful. ​

So naturally, the next logical step is to leverage the hardware implant to do something that makes it "remotely" accessible. 

The best channel to allow remote access to an attacker tends to generally be via the very networks the target device is attached to. (Which is actually what our product does: find anomalies or changes in the way a device communicates, hinting at something suspicious). The most obvious way for a hardware implant to accomplish this to leverage some kind of "software component" to do things like:

A "SoC" (System On Chip) which contains many little chips inside it (often by different manufacturers).

1. Inject attackers code into a running computer 
2. Allow remote access for the attacker via some network
3. ​Allow an attacker (masquerading as an average user of that computer) to do more than their user is supposed to​

Categories of Hardware Implants...

For all the above reasons you tend to have different types of hardware implants that leverage varying degrees of "software" interaction and consequently have different use-cases, requiring different levels-of-effort from the attacker. We categorize them as follows:

1. External Standalone Implants (A packet sniffer device listening to your ethernet or wifi data, or camera/microphone spying on keystrokes)
2. External-Peripheral Implants (A USB keystroke logger physically plugged into your computer)
3. Internal-Peripheral Implants (a device added to the assembly of a device, physically integrated with the PCB)
4. ​Internal Implants (internal to the logic of a chip, more on this later.)
5. Software (Firmware) Implants (backdoors, or "bugdoors", implanted in the software/firmware of a device)

Much like the "home spy" analogy earlier in this article, each of these categories of devices has their own "operational cost", and their operational utility hinges entirely on the operational and environmental constraints of the target. But here are some real-world examples of categories of the above devices.

1. External Standalone Implants

Packet sniffers and devices like PwnPlugs are perhaps the best examples of these kinds of devices. They are like microphones or cameras in that they are self-contained. In fact, there was a reddit post about a week ago where such a device was implanted at a user's home. There have also been more interesting highly technical devices like this that are standalone that listen to audio of keystrokes, perform power analysis to extract data, or use tiny fluctuations in Wifi signal (backscatter) to "see" into a room or even transmit data. 

There has been a plethora of these kinds of interesting "covert channel" eavesdropping research from the really talented programs at Ben-Gurion University in Israel.

---

2. External-Peripheral Implants

Devices like the USB Rubber Ducky (which automates keystrokes as if an attacker were typing), keystroke loggers (pictured to the left) are "implants" that act as peripherals to the target device. You can argue that ye olde DMA attacks like those that leveraged Firewire and the newer Thunderbolt attacks (demonstrated on MacBooks) are an example of External-Peripheral Implants. Even the recent "Graybox" iPhone attacks can be argued to be a kind of "External Peripheral Implant".

GrayBox device used to unlock iphones.

---

3. Internal-Peripheral Implants

This category of device is also what we believe the SuperMicro implant to be...

An XBox Mod-Chip

This the category of implants are where they start to get more technical (and for us, interesting). These implant go into the device, often being soldered directly to the circuit board. Less nefarious kinds of implants like this would be an XBox Mod Chip. This category of implants seem the most effective, because they would likely go un-noticed by the end-user of the device, and can be installed AFTER the manufacturing of the device.

ArsTechnica broke a story some years ago about the NSA being allegedly involved in performing these kinds of implants "during shipping" of devices. This category of device is also what we believe the SuperMicro implant to be based on the news coverage.

One common revelation among engineers learning more about hardware is that circuit boards and chips have common interfaces that form little "networks" down in the circuit. The chips all talk to each other via these interfaces. These interfaces (when tapped into) can allow for everything from code injection, to memory reading, and OCD (on-chip debugging) control of how a processor executes.
​

Fun Fact: VGA and HDMI cables even have serial communication busses built into them. We built a rig to sniff this data.

Multiple chips can be "networked" together in a circuit board using JTAG. These networks are similar to the old Token Ring networks you may remember if you are old enough.

"JTAG Blackbox Reverse Engineering" paper by Felix Domke

The aforementioned is generally achieved via JTAG. Which you may have heard about. We spend a LOT of time explaining JTAG to manufacturers and security people, because there has been very little solid public information about it. In fact, our blogpost here: "JTAG Explained" does several thousand inbound links per week from search engines. It is scary that something used in virtually every chip was so poorly understood. But this is changing. 

How usable are these kinds of interfaces for implants? Extremely. Some of the documents leaked about the NSA cataloged devices that used JTAG interfaces in servers to inject code and read/extract data.

And often times, the hardware doesnt need to be heavily modified to achieve this. There is a great talk by Felix Domke called "JTAG Blackbox Reverse Engineering" (youtube) in which he discovers undocumented functionality in a device that yields arbitrary read/write access to system memory. Thereby allowing for bypass of OS security mechanisms. 

---

​4. Internal Implants

A "SoC" (System On Chip) which contains many little chips inside it (often by different manufacturers).

We spend a lot of time talking about this in our trainings, because they are the most interesting example of the Supply-Chain issue. Most modern processors have MANY MANY "cores" inside a single chip. In fact, in the IC (Integrated Circuit) chips that power cellphones called "SoCs" (System On Chip), the single chip actually has EVERYTHING that the device needs...in a one single chip: from USB and audio, to video, ethernet, and memory cards. All handled by one chip (pictured to the left).

So WITHIN a single chip, tens or hundreds of vendors and manufacturers can have their "chips"

Manufacturers increasingly don't manufacture every single sub-component. Instead they purchase or license the designs from another company. These designs come in the form of an "RTL" or design file that describe how the silicon and logic of that core may be manufactured. So WITHIN a single chip, tends or hundreds of vendors and manufacturers can have their chips. This is in fact how ARM works (ARM does not make chips, unlike Intel and AMD.  Instead, they just license the designs to companies like Apple, Microchip, et al who manufacture the physical chips). Some might argue that this is actually the secret to their incredible market-share.

Editing an RTL in Synopsys ASIC design drafting software

These "RTL"s are literally like software libraries are to software developers. Similar to the way a software developer imports libraries to help them write their program (network libraries, protocol libraries, file format output libraries, etc), chip designers do the same thing.  They use CAD and design software to build chips and PCBs (like a highly technical version of "Visio"). This design software allows the designer to "import" entire sections of designs to use in their project.  In some cases, importing these libraries  is literally a "drag and drop" the way you drag images or "shapes" into your Visio or PowerPoint project. 

Due to this model, hardware validation or "formal verification" has arisen as a way to algorithmically (or exhaustively) prove that a hardware implementation doesnt do more than it was designed to do, or that it does not have extra undocumented features (aka backdoors). Companies like Synopsys or our friends at Galois make fancy hardware emulation software or algorithmic provers that chew through chip designs looking for implementation deviations. In fact, former Xipiter employee and great hardware security trainer Joe Fitzpatrick (mentioned in the Bloomberg article above and contributor to the NSA Playset) used to do this professionally. 

From the "JTAG Attacks And Defenses" paper from NYU Polytechnic 2009​

This is obviously the most subversive and covert kind of implant, because it lives in the chip beyond the scrutiny of even the most paranoid end user. But obviously the cost performing this kind of intentional backdooring is much higher. A somewhat more feasible technique is to discover some undocumented functionality in a subcore/component (ala "Felix Domke" above) that allows for a knowledgable attacker to leverage this functionality to achieve their goals. There have been many suspicions that the Intel and AMD hardware "cheat codes" of recent years (including some surprisingly lesser publicized AMD gaffes dating as far back as 2004) may have been something like this. This is a great little article by Dan Luu discussing the feasibility of CPU backdoors.

---

5. Software (Firmware) Implants

These are probably better known, due to all the public examples (like the Sony/BMG backdoor et. al) but in our opinion, most hardware implants are likely attempting to facilitate or perform some modicum of software backdooring. Conversely, "pure" software implants, require no hardware. In our hardware hacking courses we have participants use their hardware level access to modify the of a device enabling them to remotely access the system via the network. We mention this, because this kind of firmware backdooring is EXTREMELY easy to do on many consumer devices. So much so that it is a teachable exercise within a couple hours. With the right backdoor, you can even exfiltrate data in interesting ways (like via the LEDs). We demonstrated a method for that along with the remote vulnerability we disclosed in a router with this blopost. 

This kind of firmware backdooring is EXTREMELY easy to do on many devices.

We suspect that the SuperMicro backdoor didnt do this directly but instead leveraged the nature of the device meant to fetch firmware from a specific piece of hardware. (more later) ​​

So what do we think this thing actually is?

Credit: Bloomberg article

There is very little in tractable technical detail in all the public articles about this incident. This doesn't seem to be a deceptive omission, but instead an attempt to make it more consumable by the layperson. We immediately suspected it was "Category 3" implant (aka "Internal-Peripheral Implant") with maybe some components of "Category 5" (aka "Software/Firmware Implant"). Much of the Bloomberg and ArsTechnica articles seem to support that.

Our favorite bit of "evidence" comes from Trammell Hudson who does some fantastic work on firmware and backdoors. In his short tweet, he noticed that the location that the backdoor chip was placed resides at the 8-pin SOIC (Single Outline IC) where a chip was omitted by the manufacturer.

For context here, 8-pin SOIC chips are often used for EEPROM, a place to store firmware that is loaded by the device at boot time. (We discuss this in great detail as it relates to a router here. And more about SPI EEPROM in this video by one team members.)

The reason this possibility is compelling is that it is the most likely place for a "Class 3 implant" to masquerade as the original EEPROM chip (which contains a modified version of some firmware) which will in turn be loaded or used by the device. There a "Class 5" firmware implant is waiting to begin doing what it needs to do to allow remote access or "spy" on the target device.

We also suspect, that the way the story is being reported is actually conflating several findings about the role of the hardware backdoor. It seems more technically feasible that the hardware implant is "bootstrapping" or injecting code that then handles the  kernel or userland parts of the remote access. 

Another very fascinating datapoint is the work from the folks at Eclypsium SPECIFICALLY on the vulnerabilities of the SuperMicro firmware update mechanisms that are being called into question in the Bloomberg article. This work was published in early September well in advance of the Bloomberg publication. 

Credit: Eclypsium

The nature of the vulnerabilities highlighted in Eclypsium's work, could potentially be exploited/leveraged if the firmware was bootstrapped from a EEPROM chip like that mentioned by Trammell Hudson above.

Conclusions

Hardware backdoors are mostly useless (or at least operationally expensive) without some component of software. We dont need to be too worried about these kinds of implants being used pervasively to "mass target", but they are a reminder of the "Art of the Possible". And certainly for institutions with "something to lose" targeted hardware implants are a real threat. But for the average person or institution, Class 1, 2, (and Class 5) implants are more likely to be what they're up against.  

The broader issue that this should also raise is: "How real or prevalent are 'Class 4' implants?" That is to say, how many devices do we CURRENTLY use today have undocumented functionality baked into that hardware that is already potentially being used by savvy actors? 

Note: We think about this stuff A LOT, and have a lot more to add so we will likely continuously add to this blogpost over the next few days. We will note revisions here:

​Update 10:21am PST 5Oct2018 (added quite a few links/references to articles mentioned). Added new section, new images. Grammar corrections.

Update 17:32am PST 5Oct2018 (added references to Eclypsium's work)