The Big Hack: October 8, 2018
The Big Hack: October 8, 2018
The Big Hack: October 8, 2018
▹ Citizen Strong’s
John Burton left a job in
banking after Trump’s
2016 election win
5
PHOTOGRAPH BY MICHAEL FRIBERG FOR BLOOMBERG BUSINESSWEEK
51
n 2015, Amazon.com Inc. began quietly evaluating a startup familiar with the matter say investigators found that the chips
I called Elemental Technologies, a potential acquisition to had been inserted at factories run by manufacturing subcon-
help with a major expansion of its streaming video service, tractors in China.
known today as Amazon Prime Video. Based in Portland, Ore., This attack was something graver than the software-based
Elemental made software for compressing massive video files incidents the world has grown accustomed to seeing.
and formatting them for different devices. Its technology had Hardware hacks are more difficult to pull off and potentially
helped stream the Olympic Games online, communicate with more devastating, promising the kind of long-term, stealth
the International Space Station, and funnel drone footage to access that spy agencies are willing to invest millions of dol-
the Central Intelligence Agency. Elemental’s national security lars and many years to get.
contracts weren’t the main reason for the proposed acquisi- There are two ways for spies to alter the guts of computer
52 tion, but they fit nicely with Amazon’s government businesses, equipment. One, known as interdiction, consists of manip-
such as the highly secure cloud that Amazon Web Services ulating devices as they’re in transit from manufacturer to
(AWS) was building for the CIA. customer. This approach is favored by U.S. spy agencies,
To help with due diligence, AWS, which was overseeing the according to documents leaked by former National Security
prospective acquisition, hired a third-party company to scru- Agency contractor Edward Snowden. The other method
tinize Elemental’s security, according to one person familiar involves seeding changes from the very beginning.
with the process. The first pass uncovered troubling issues, One country in particular has an advantage executing this
prompting AWS to take a closer look at Elemental’s main prod- kind of attack: China, which by some estimates makes 75 per-
uct: the expensive servers that customers installed in their net- cent of the world’s mobile phones and 90 percent of its PCs.
works to handle the video compression. These servers were Still, to actually accomplish a seeding attack would mean devel-
assembled for Elemental by Super Micro Computer Inc., a San oping a deep understanding of a product’s design, manipulat-
Jose-based company (commonly known as Supermicro) that’s ing components at the factory, and ensuring that the doctored
also one of the world’s biggest suppliers of server mother- devices made it through the global logistics chain to the desired
boards, the fiberglass-mounted clusters of chips and capaci- location—a feat akin to throwing a stick in the Yangtze River
tors that act as the neurons of data centers large and small. In upstream from Shanghai and ensuring that it washes ashore
late spring of 2015, Elemental’s staff boxed up several servers in Seattle. “Having a well-done, nation-state-level hardware
and sent them to Ontario, Canada, for the third-party security implant surface would be like witnessing a unicorn jumping
company to test, the person says. over a rainbow,” says Joe Grand, a hardware hacker and the
Nested on the servers’ motherboards, the testers found founder of Grand Idea Studio Inc. “Hardware is just so far off
a tiny microchip, not much bigger than a grain of rice, that the radar, it’s almost treated like black magic.”
wasn’t part of the boards’ original design. Amazon reported But that’s just what U.S. investigators found: The chips had
the discovery to U.S. authorities, sending a shudder through been inserted during the manufacturing process, two officials
the intelligence community. Elemental’s servers could be say, by operatives from a unit of the People’s Liberation Army.
found in Department of Defense data centers, the CIA’s drone In Supermicro, China’s spies appear to have found a perfect
operations, and the onboard networks of Navy warships. And conduit for what U.S. officials now describe as the most sig-
Elemental was just one of hundreds of Supermicro customers. nificant supply chain attack known to have been carried out
During the ensuing top-secret probe, which remains open against American companies.
more than three years later, investigators determined that the One official says investigators found that it eventually
chips allowed the attackers to create a stealth doorway into any affected almost 30 companies, including a major bank,
network that included the altered machines. Multiple people government contractors, and the world’s most valuable
Bloomberg Businessweek October 8, 2018
found in made-to-order server setups at banks, hedge funds, when Amazon made its discovery and gave them access to sab-
cloud computing providers, and web-hosting services, among otaged hardware, according to one U.S. official. This created
other places. Supermicro has assembly facilities in California, an invaluable opportunity for intelligence agencies and the
the Netherlands, and Taiwan, but its motherboards—its core FBI—by then running a full investigation led by its cyber- and
product—are nearly all manufactured by contractors in China. counterintelligence teams—to see what the chips looked like
The company’s pitch to customers hinges on unmatched and how they worked.
customization, made possible by hundreds of full-time engi- The chips on Elemental servers were designed to be as
neers and a catalog encompassing more than 600 designs. The inconspicuous as possible, according to one person who saw
majority of its workforce in San Jose is Taiwanese or Chinese, a detailed report prepared for Amazon by its third-party secu-
and Mandarin is the preferred language, with hanzi filling the rity contractor, as well as a second person who saw digital
whiteboards, according to six former employees. Chinese pas- photos and X-ray images of the chips incorporated into a later
tries are delivered every week, and many routine calls are done report prepared by Amazon’s security team. Gray or off-white
twice, once for English-only workers and again in Mandarin. in color, they looked more like signal conditioning couplers,
The latter are more productive, according to people who’ve another common motherboard component, than microchips,
been on both. These overseas ties, especially the widespread and so they were unlikely to be detectable without specialized
use of Mandarin, would have made it easier for China to gain equipment. Depending on the board model, the chips varied
an understanding of Supermicro’s operations and potentially slightly in size, suggesting that the attackers had supplied dif-
to infiltrate the company. (A U.S. official says the government’s ferent factories with different batches.
probe is still examining whether spies were planted inside Officials familiar with the investigation say the primary role
Supermicro or other American companies to aid the attack.) of implants such as these is to open doors that other attackers
With more than 900 customers in 100 countries by 2015, can go through. “Hardware attacks are about access,” as one
Supermicro offered inroads to a bountiful collection of sen- former senior official puts it. In simplified terms, the implants
sitive targets. “Think of Supermicro as the Microsoft of the on Supermicro hardware manipulated the core operating
hardware world,” says a former U.S. intelligence official instructions that tell the server what to do as data move across
who’s studied Supermicro and its business model. “Attacking a motherboard, two people familiar with the chips’ operation
54 Supermicro motherboards is like attacking Windows. It’s like say. This happened at a crucial moment, as small bits of the
attacking the whole world.” operating system were being stored in the board’s temporary
memory en route to the server’s central processor, the CPU.
ell before evidence of the attack surfaced inside the The implant was placed on the board in a way that allowed it to
W networks of U.S. companies, American intelligence effectively edit this information queue, injecting its own code
sources were reporting that China’s spies had plans or altering the order of the instructions the CPU was meant to
to introduce malicious microchips into the supply chain. The follow. Deviously small changes could create disastrous effects.
sources weren’t specific, according to a person familiar with Since the implants were small, the amount of code they
the information they provided, and millions of motherboards contained was small as well. But they were capable of doing
are shipped into the U.S. annually. But in the first half of 2014, two very important things: telling the device to communicate
a different person briefed on high-level discussions says, intel- with one of several anonymous computers elsewhere on the
ligence officials went to the White House with something more internet that were loaded with more complex code; and pre-
concrete: China’s military was preparing to insert the chips into paring the device’s operating system to accept this new code.
Supermicro motherboards bound for U.S. companies. The illicit chips could do all this because they were connected
The specificity of the information was remarkable, but to the baseboard management controller, a kind of superchip
so were the challenges it posed. Issuing a broad warning to that administrators use to remotely log in to problematic serv-
Supermicro’s customers could have crippled the company, a ers, giving them access to the most sensitive code even on
major American hardware maker, and it wasn’t clear from the machines that have crashed or are turned off.
intelligence whom the operation was targeting or what its ulti- This system could let the attackers alter how the device
mate aims were. Plus, without confirmation that anyone had functioned, line by line, however they wanted, leaving no one
been attacked, the FBI was limited in how it could respond. the wiser. To understand the power that would give them, take
The White House requested periodic updates as information this hypothetical example: Somewhere in the Linux operating
came in, the person familiar with the discussions says. system, which runs in many servers, is code that authorizes a
Apple made its discovery of suspicious chips inside user by verifying a typed password against a stored encrypted
Supermicro servers around May 2015, after detecting odd net- one. An implanted chip can alter part of that code so the server
work activity and firmware problems, according to a person won’t check for a password—and presto! A secure machine is
familiar with the timeline. Two of the senior Apple insiders open to any and all users. A chip can also steal encryption keys
say the company reported the incident to the FBI but kept for secure communications, block security updates that would
details about what it had detected tightly held, even internally. neutralize the attack, and open up new pathways to the inter-
Government investigators were still chasing clues on their own net. Should some anomaly be noticed, it would likely be
Bloomberg Businessweek October 8, 2018
55
he compromised
motherboards were built
into servers assembled
by Supermicro.
④ The sabotaged servers
made their way inside data
centers operated by dozens
of companies.
been found, according to a person familiar with the com- conference headlined by a landmark deal on cybersecu-
pany’s probe. Instead, the team developed a method of mon- rity. After months of negotiations, the U.S. had extracted
itoring the chips. In the ensuing months, they detected brief from China a grand promise: It would no longer support
check-in communications between the attackers and the the theft by hackers of U.S. intellectual property to bene-
sabotaged servers but didn’t see any attempts to remove fit Chinese companies. Left out of those pronouncements,
data. That likely meant either that the attackers were sav- according to a person familiar with discussions among senior
ing the chips for a later operation or that they’d infiltrated officials across the U.S. government, was the White House’s
other parts of the network before the monitoring began. deep concern that China was willing to offer this concession
Neither possibility was reassuring. because it was already developing far more advanced and
When in 2016 the Chinese government was about to pass surreptitious forms of hacking founded on its near monop-
a new cybersecurity law—seen by many outside the country oly of the technology supply chain.
as a pretext to give authorities wider access to sensitive data— In the weeks after the agreement was announced, the U.S.
Amazon decided to act, the person familiar with the company’s government quietly raised the alarm with several dozen tech
probe says. In August it transferred operational control of its executives and investors at a small, invite-only meeting in
Beijing data center to its local partner, Beijing Sinnet, a move McLean, Va., organized by the Pentagon. According to some-
the companies said was needed to comply with the incoming one who was present, Defense Department officials briefed
law. The following November, Amazon sold the entire infra- the technologists on a recent attack and asked them to think
structure to Beijing Sinnet for about $300 million. The per- about creating commercial products that could detect hard-
son familiar with Amazon’s probe casts the sale as a choice to ware implants. Attendees weren’t told the name of the hard-
“hack off the diseased limb.” ware maker involved, but it was clear to at least some in the
As for Apple, one of the three senior insiders says that room that it was Supermicro, the person says.
in the summer of 2015, a few weeks after it identified The problem under discussion wasn’t just technological.
the malicious chips, the company started removing all It spoke to decisions made decades ago to send advanced
Supermicro servers from its data centers, a process Apple production work to Southeast Asia. In the intervening years,
referred to internally as “going to zero.” Every Supermicro low-cost Chinese manufacturing had come to underpin the
58 server, all 7,000 or so, was replaced in a matter of weeks, the business models of many of America’s largest technology com-
senior insider says. (Apple denies that any servers were panies. Early on, Apple, for instance, made many of its most
removed.) In 2016, Apple informed Supermicro that it was sophisticated electronics domestically. Then in 1992, it closed
severing their relationship entirely—a decision a spokes- a state-of-the-art plant for motherboard and computer assem-
man for Apple ascribed in response to Businessweek’s ques- bly in Fremont, Calif., and sent much of that work overseas.
tions to an unrelated and relatively minor security incident Over the decades, the security of the supply chain became
(sidebar, page 57). an article of faith despite repeated warnings by Western offi-
That August, Supermicro’s CEO, Liang, revealed that the cials. A belief formed that China was unlikely to jeopardize
company had lost two major customers. Although he didn’t its position as workshop to the world by letting its spies med-
name them, one was later identified in news reports as Apple. dle in its factories. That left the decision about where to build
He blamed competition, but his explanation was vague. “When commercial systems resting largely on where capacity was
customers asked for lower price, our people did not respond greatest and cheapest. “You end up with a classic Satan’s bar-
quickly enough,” he said on a conference call with analysts. gain,” one former U.S. official says. “You can have less supply
Hayes, the Supermicro spokesman, says the company has than you want and guarantee it’s secure, or you can have the
never been notified of the existence of malicious chips on its supply you need, but there will be risk. Every organization
motherboards by either customers or U.S. law enforcement. has accepted the second proposition.”
Concurrent with the illicit chips’ discovery in 2015 and In the three years since the briefing in McLean, no
the unfolding investigation, Supermicro has been plagued commercially viable way to detect attacks like the one on
by an accounting problem, which the company characterizes Supermicro’s motherboards has emerged—or has looked
as an issue related to the timing of certain revenue recogni- likely to emerge. Few companies have the resources of Apple
tion. After missing two deadlines to file quarterly and annual and Amazon, and it took some luck even for them to spot the
reports required by regulators, Supermicro was delisted from problem. “This stuff is at the cutting edge of the cutting edge,
the Nasdaq on Aug. 23 of this year. It marked an extraordi- and there is no easy technological solution,” one of the peo-
nary stumble for a company whose annual revenue had risen ple present in McLean says. “You have to invest in things that
sharply in the previous four years, from a reported $1.5 bil- the world wants. You cannot invest in things that the world
lion in 2014 to a projected $3.2 billion this year. is not ready to accept yet.” 0
Bloomberg LP has been a Supermicro customer. According to
ne Friday in late September 2015, President Barack a Bloomberg LP spokesperson, the company has found no evi-
O Obama and Chinese President Xi Jinping appeared dence to suggest that it has been affected by the hardware issues
together at the White House for an hourlong press raised in the article.
Bloomberg Businessweek October 8, 2018
Statements
Amazon fully mitigated by the auditors if purchased from Super Micro constantly responding to threats
customers used the appliances as when we updated the firmware and evolving their security
It’s untrue that AWS knew about intended, without exposing them and software according to our posture. As part of that effort
a supply chain compromise, an to the public internet. standard procedures. we are in regular contact with
issue with malicious chips, or Additionally, in June 2018, We are deeply disappointed a variety of vendors, industry
hardware modifications when researchers made public reports that in their dealings with us, partners and government
acquiring Elemental. It’s also of vulnerabilities in SuperMicro Bloomberg’s reporters have not agencies sharing information on
untrue that AWS knew about firmware. As part of our standard been open to the possibility that threats, best practices and new
servers containing malicious operating procedure, we notified they or their sources might be tools. This is standard practice in
chips or modifications in data affected customers promptly, and wrong or misinformed. Our best the industry today. However, we
centers based in China, or that recommended they upgrade the guess is that they are confusing have not been in contact with any
AWS worked with the FBI to firmware in their appliances. their story with a previously- government agency regarding the
investigate or provide data about reported 2016 incident in which issues you raised.
malicious hardware. we discovered an infected driver Furthermore, Supermicro
We’ve re-reviewed our Apple on a single Super Micro server doesn’t design or manufacture
records relating to the Elemental in one of our labs. That one-time networking chips or the
acquisition for any issues related Over the course of the past event was determined to be associated firmware and we,
to SuperMicro, including re- year, Bloomberg has contacted accidental and not a targeted as well as other leading server/
examining a third-party security us multiple times with claims, attack against Apple. storage companies, procure them
audit that we conducted in 2015 sometimes vague and sometimes While there has been no from the same leading networking
as part of our due diligence prior elaborate, of an alleged security claim that customer data was companies.
to the acquisition. We’ve found incident at Apple. Each time, we involved, we take these allegations
no evidence to support claims have conducted rigorous internal seriously and we want users
of malicious chips or hardware investigations based on their to know that we do everything China’s Ministry of
modifications. inquiries and each time we have possible to safeguard the personal Foreign Affairs
The pre-acquisition audit found absolutely no evidence to information they entrust to us.
described four issues with a support any of them. We have We also want them to know that China is a resolute defender 59
web application (not hardware repeatedly and consistently what Bloomberg is reporting about of cybersecurity. It advocates
or chips) that SuperMicro offered factual responses, on the Apple is inaccurate. for the international community
provides for management of their record, refuting virtually every Apple has always believed in to work together on tackling
motherboards. All these findings aspect of Bloomberg’s story being transparent about the ways cybersecurity threats through
were fully addressed before we relating to Apple. we handle and protect data. If dialogue on the basis of mutual
acquired Elemental. The first two On this we can be very clear: there were ever such an event as respect, equality and mutual
issues, which the auditor deemed Apple has never found malicious Bloomberg News has claimed, benefit.
as critical, related to a vulnerability chips, “hardware manipulations” or we would be forthcoming about Supply chain safety in
in versions prior to 3.15 of this web vulnerabilities purposely planted it and we would work closely with cyberspace is an issue of common
application (our audit covered prior in any server. Apple never had any law enforcement. Apple engineers concern, and China is also a
versions of Elemental appliances contact with the FBI or any other conduct regular and rigorous victim. China, Russia, and other
as well), and these vulnerabilities agency about such an incident. We security screenings to ensure member states of the Shanghai
had been publicly disclosed are not aware of any investigation that our systems are safe. We Cooperation Organization
by SuperMicro on 12/13/2013. by the FBI, nor are our contacts in know that security is an endless proposed an “International
Because Elemental appliances are law enforcement. race and that’s why we constantly code of conduct for information
not designed to be exposed to the In response to Bloomberg’s fortify our systems against security” to the United Nations
public internet, our customers are latest version of the narrative, we increasingly sophisticated hackers as early as 2011. It included a
protected against the vulnerability present the following facts: Siri and cybercriminals who want to pledge to ensure the supply
by default. Nevertheless, the and Topsy never shared servers; steal our data. chain security of information
Elemental team had taken the Siri has never been deployed on and communications technology
extra action on or about 1/9/2014 servers sold to us by Super Micro; products and services, in order to
to communicate with customers and Topsy data was limited to Supermicro prevent other states from using
and provide instructions to approximately 2,000 Super Micro their advantages in resources and
download a new version of the servers, not 7,000. None of those While we would cooperate with technologies to undermine the
web application from SuperMicro servers has ever been found to any government investigation, we interest of other countries. We
(and after 1/9/2014, all appliances hold malicious chips. are not aware of any investigation hope parties make less gratuitous
shipped by Elemental had updated As a matter of practice, before regarding this topic nor have accusations and suspicions but
versions of the web application). servers are put into production we been contacted by any conduct more constructive talk
So, the two “critical” issues that at Apple they are inspected for government agency in this regard. and collaboration so that we
the auditor found, were actually security vulnerabilities and we We are not aware of any customer can work together in building a
fixed long before we acquired update all firmware and software dropping Supermicro as a supplier peaceful, safe, open, cooperative
Elemental. The remaining two with the latest protections. We for this type of issue. and orderly cyberspace.
non-critical issues with the web did not uncover any unusual Every major corporation —Translated by Bloomberg News
application were determined to be vulnerabilities in the servers we in today’s security climate is in Beijing