GitHub Disrupted by 1.5 Tbps Onslaught as New DDoS Attack Type Emerges
If 2018 hasn’t been exciting enough yet, say hello to a new type of distributed denial-of-service attack: UDP amplification via servers running memcached, an open source distributed caching system.
See Also: How to Scale Your Vendor Risk Management Program
Proof of the attack’s efficacy came via disruptions of multiple sites, with security researchers seeing a DDoS attack that hit GitHub on Wednesday peak at a massive 1.35 terabits per second, followed by a second attack peaking at 400 Gbps.
A report from GitHub, a popular code-sharing website based in San Francisco, says its site was unavailable or intermittently available for only 10 minutes. This wasn’t the first time GitHub had been targeted by online disruptors, but it was one of the largest DDoS attacks that has been seen to date (see GitHub DDoS Attack Traces to China).
Making Memcached Attack
Here’s how memcached servers can be abused: DDoS attacks typically disrupt servers either by overwhelming them with sheer volumes of traffic or else targeting specific services, which can become choke points when over-queried. Such attacks often involve sending a query to a site and then reflecting it onto the targeted site.
Enter the memcached protocol, which by default listens on TCP and UDP port 11211 on most versions of Linux and BSD.
Last week, security researchers began spotting attacks that exploit this obscure service to overwhelm websites with massive quantities of IP traffic. DDoS defense firm Akamai reported seeing multiple attacks, including one on Wednesday against its customer GitHub that hit 1.3 terabits per second.
“Memcached is a tool meant to cache data and reduce strain on heavier data stores, like disk or databases. The protocol allows the server to be queried for information about key value stores and is only intended to be used on systems that are not exposed to the internet,” Akamai says.
Servers running the protocol listen on UDP port 11211 by default. But when servers running the protocol are exposed to the internet, memcached “can be exploited to produce DDoS amplification attacks by sending the memcached server a UDP packet with a spoofed IP containing a message asking for statistics, which will cause the server to return an enormous message to the victim,” Imperva Incapsula’s Avishay Zawoznik, a security research team leader, and Johnathan Azaria, a security researcher, write in a blog post.
As of last week, more than 90,000 endpoints running memcached were exposed to the internet, according to internet of things search engine Shodan.
An update for memcached has been released that disables UDP by default. But it’s not clear how many users have downloaded and installed the update.
Until a server running memcached gets updated, if it’s internet-connected, it can be abused to launch massive DDoS attacks. “There is no authentication required with memcached,” Akamai says. “When this is added to the ability to spoof IP addresses of UDP traffic, the protocol can be easily abused as a reflector when it is exposed to the internet.”
Pump It Up
Attackers have been using memcached to amplify their attacks. This means they can send a small query and get a big response back, which for attackers equals more firepower.
“The technical name for this is a ‘UDP-based reflection attack vector,’ which is just a fancy way of saying, ‘We’re going to bury your server under a thousand miles of data-driven concrete’,” says Christopher Boyd, lead malware intelligence analyst at Malwarebytes, in a blog post.
“Memcached can have an amplification factor of over 50,000, meaning a 203 byte request results in a 100 megabyte response,” Akamai says.
Multiple DDoS defense firms have warned that simply blocking port 11211 will not mitigate memcached abuse.
“Although the attack can be easily detected by setting a rule on UDP traffic coming from source port 11211, it cannot be mitigated without a dedicated DDoS mitigation solution due to the massive amounts of PPS [packets per second] and BPS [bits per second] delivered on such an attack, which are likely to turn edge routing devices unavailable before traffic arrives at the server, regardless of its configuration,” Imperva says.
On-demand DDoS attacks can be procured on underground forums, where they’re often marketed as “stresser/booter” services, a quasi-legitimate spin suggesting that the services are only to be used to legitimately stress test networks when they receive overwhelming amounts of traffic.
In reality, law enforcement experts say bad actors regularly use stresser/booter services to target legitimate services (see Lloyds Banking Group Reportedly Hit by DDoS Attack).
Top DDoS-Attacked Industries
It’s rare for a major new type of DDos attack to emerge. “A discovery of a new amplification vector, though allowing very great amplification, happens rarely,” DDoS defense firm CloudFlare says in a blog post. “This new memcached UDP DDoS is definitely in this category.”
Developers Update Memcached
Thankfully, there’s an easy fix for these attacks, at least if administrators are proactive.
“Similar to most reflection and amplification attacks before it, the primary solution to memcached attacks is to not have the reflectors exposed to the internet,” Akamai says. “However, relying on remote systems administrators to remove their servers from the internet is not a solution likely to see immediate results.”
The developers behind memcached say that the latest version, Memcached 1.5.6, released Tuesday, “disables the UDP protocol by default.”
One of the developers behind memcached, Denis Anikin (@Dormando), says that UDP support was enabled 12 years ago when UDP was more widely used because it has less overhead than TCP.
probably a good time to ask; does anyone actually use the UDP interface for memcached? Last I talked to anyone who did was ten years ago. I’m thinking about gating it behind a compile flag and warning distro’s to not re-enable.
— dormando (@dormando) March 1, 2018
“In the last few years, I’ve not heard of anyone using UDP anymore,” Anikin tweets. “That said, I don’t have any way of knowing how many UDP installations there are.” As a result, UDP is disabled by default unless whoever installs it explicitly enables it.
“Hopefully this one-time pain is acceptable,” he adds. And hopefully, everyone using memcached rolls out updates as quickly as possible.