How whitehats stopped the DDoS attack that knocked Spamhaus offline

As an international organization that disrupts spam operators, the Spamhaus Project has made its share of enemies. Many of those enemies possess the Internet equivalent of millions of water cannons that can be turned on in an instant to flood targets with more traffic than they can possibly stand.

On Tuesday, Spamhaus came under a torrential deluge—75 gigabits of junk data every second—making it impossible for anyone to access the group's website (the real-time blacklists that ISPs use to filter billions of spam messages were never effected). Spamhaus quickly turned to CloudFlare, a company that secures websites and helps mitigate the effects of distributed denial-of-service attacks.

This is a story about how the attackers were able to flood a single site with so much traffic, and the way CloudFlare blocked it using a routing methodology known as Anycast.

While attacks of 100Gbps aren't unheard of, the 75Gbps assault was still massive and generally well beyond what most botnets are capable of generating. To magnify their limited amount of bandwidth, the attackers resorted to what's known as DNS (domain name system) amplification—a technique that allows attackers to multiply their junk traffic by as much as 100 fold. As Ars explained in October, DNS amplification attacks work because companies such as AT&T, GoDaddy, SoftLayer, and Pakistan Telecom allow open DNS servers to run on the networks they operate instead of limiting them to just paying customers. DDoS attackers have abused these open DNS resolvers for years in a way that severely aggravates the effects of their crippling assaults.

As many Ars readers know, DNS servers are the Internet directories that translate domain names such as arstechnica.com into IP addresses such as 50.31.151.33. But DNS servers can also be queried for the IP addresses of huge swaths of the Internet, putting the person listed as making the request on the receiving end of a massive response. In a blog post published Wednesday, CloudFlare CEO Matthew Prince said each DNS request sent by the Spamhaus attackers was likely only 36 bytes long, while each response was about 3,000 bytes. By spoofing the requests to make them appear as if they originated with Spamhaus, the attackers can turn the firepower of all those networks against their opponent, all but guaranteeing it won't be available to process legitimate traffic.

To get Spamhaus back online, CloudFlare relied on Anycast, a routing technique that distributes the same IP address across 23 data centers across the world. Internet traffic almost always chooses the shortest physical path. Anycast allows the geographically dispersed junk traffic to be absorbed by dozens of individual centers, where each packet is then inspected. When it bears signatures found in the attack traffic—for example, if it's a 3,000-byte response from an open DNS resolver—it is discarded in the CloudFlare data center. Only Legitimate Web requests are allowed to be forwarded to the Spamhaus data center.

"When there's an attack, Anycast serves to effectively dilute it by spreading it across our facilities," Prince wrote. "Since every data center announced the same IP address for any CloudFlare customer, traffic cannot be concentrated in any one location. Instead of the attack being many-to-one, it becomes many-to-many with no single point on the network acting as a bottleneck."

Anycast made it easy for CloudFlare to filter out other types of malicious traffic directed at Spamhaus. The attackers also flooded the anti-spam service with huge numbers of spoofed packets bearing the ACK flag, which is the second part of the multi-step handshake computers on the Internet follow to establish connections.

"In an ACK reflection, the attacker sends a number of SYN packets to servers with a spoofed source IP address pointing to the intended victim," Prince wrote. "The servers then respond to the victim's IP with an ACK. Like the DNS reflection attack, this disguises the source of the attack, making it appear to come from legitimate servers."

The attacks are significantly easier to block since there's no amplification effect. CloudFlare drops each unmatched ACK.

Ironically, when CloudFlare blocks these types of attacks it routinely hears from network operators who complain that the service is attacking their systems with abusive DNS queries or SYN floods. And therein shows the work that remains to get the DoS problem under control. As effective as Anycast is at lessening the effects of denial of service attacks, it's akin to cough medicine that treats the symptom while doing nothing to cure the cold that causes it in the first place. As Ars learned first-hand last week, just about anyone can wield a DoS club that can make it impossible for legitimate traffic to get through. Ridding the Internet of the scourge will require a combination of education and pressure on network providers to prevent their infrastructure from attacking innocent bystanders.