Massive 1.7 Tbps Reflection/Amplification Attack Exploits Open Memcached Servers
Nearly 100,000 servers could still be vulnerable to being abused to launch massive distributed denial-of-service attacks.
See Also: Effective Cyber Threat Hunting Requires an Actor and Incident Centric Approach
The risk comes via Memcached, an open source utility designed to cache in RAM frequently used web pages (see Memcached Servers Deliver Amplified DDoS Attacks).
The web page caching utility was never designed to be internet-accessible and requires no authentication to access. But some administrators – as well as some Linux instances by default – have left TCP or UDP port 11211 open to internet-borne requests.
Attackers have begun demonstrating how badly servers with misconfigured Memcached can be abused.
Popular code-sharing website GitHub reported that on Feb. 28 it was hit by a massive DDoS attack that left its site unavailable or intermittently unreachable, but only for 10 minutes. Akamai, which provides DDoS defenses for GitHub, clocked the attack’s peak at a whopping 1.35 terabits per second.
On Monday, however, DDoS defense firm Arbor Networks, part of Netscout, reported that one of its customers, a U.S.-based service provider, was hit by a 1.7 Tbps reflection/amplification attack “based on the same Memcached reflection/amplification attack vector that made up the Github attack,” according to Arbor’s Carlos Morales, who says his firm successfully repelled the disruption.
Arbor says the biggest-ever previous DDoS attack that it logged was in the summer of 2016, when a Brazilian customer was hit with a 650 Gbps attack.
“The Memcached UDP protocol [is] a fine example of double trouble with potential for severe operational impact.”
“Memcached servers are now being used as reflectors/amplifiers to launch extremely high-volume UDP reflection/amplification attacks,” Morales says in a March 1 blog post.
“They are proving especially effective because Memcached servers have high-bandwidth access links and reside on networks with high-speed transit uplinks,” he adds. “This makes Memcached servers ideal for use in high-bandwidth reflection/amplification DDoS attacks.”
Johannes B. Ullrich, dean of research at SANS Technology Institute, describes the Memcached abuse as “a classic reflective amplified DDoS attack,” adding that “luckily, it isn’t too hard to block.”
Here’s how: “You should see traffic *from* port 11211 if you are hit by this attack,” he says ina blog post. “Blocking all traffic from port 11211 should be possible as all modern operating systems tend to use a source port higher than that for client connections. But given the traffic volumes people are seeing, you will likely need help “upstream” or from an anti-DDoS company.”
Some DDoS extortionists are now threatening to unleash Memcached-enabled disruptions on victims unless they pay ransoms in the 50 monero – aka XMR – cryptocurrency, worth about $16,000. But Ullrich says all attacks seen to date have directed victims to pay into a single monero wallet address, meaning that these opportunistic criminals would have no way to verify which victims paid or did not pay.
The Power of Amplification
Memcached offers massive bang for attackers’ buck. For starters, it can be abused to inject massive keys – up to 50 kilobytes – and attackers can then use IP spoofing to create DDoS attacks with a massive rate of amplification.
The U.S. Computer Emergency Readiness Team has updated its alert on UDP-based amplification attacks, originally released in 2014, to include Memcached-based reflection DDoS attacks via UDP/TCP port 11211. It warns that abusing Memcached can result in attacks being amplified by 10,000 to 51,000 times.
Memcached Users: Upgrade Now
As Information Security Media Group reported on Monday, version 1.5.6 of Memcached, released Feb. 27, disables UDP by default, according to Memcached’s developer, who goes by Dormando. “I’m surprised it took this long for someone to abuse it :/,” Dormando tweets. “Please upgrade, or disable UDP. … Please do not expose cache to internet, and so on.”
For what it’s worth, if you’re getting attacked by memcached’s, it’s pretty easy to disable them since the source won’t be spoofed. They may accept “shutdownrn”, but also running “flush_allrn” in a loop will prevent amplification.
— dormando (@dormando) February 27, 2018
Potential Risks, Defenses
Potentially, Memcached be used for more than DDoS attacks. On Wednesday, DDoS defense firm Corero Network Security warned that servers with open Memcached ports could potentially be abused by attackers to “reveal the ‘keys’ to your data” by using debug commands, as well as potentially to maliciously modify the data and reinsert it into the cache without the knowledge of the Memcached owner.”
“If you have a Memcached server in your environment that is exposed to the internet, then you should stop scanning for them, and spend your time writing a resume instead.”
Corero also repeated a suggestion, first aired by Dormando on Feb. 27, that sending a “flush_all” command back to any attacking server could help mitigate attacks. Doing so “invalidates a vulnerable server’s cache, including the large, potentially malicious payload planted there by attackers,” according to Corero, which suggested this maneuver might function as the equivalent of a “kill switch.”
But Mounir Hahad, head of threat research at security hardware manufacturer Juniper Networks, tells ISMG via email that he believes this defense is no silver bullet.
“This kill switch is unfortunately just a Band-Aid,” Hahad says. “It will take an attacker only a few minutes to overcome by replanting the large file in the Memcached servers again. It becomes a game of whack-a-mole: the attacker replants the file repeatedly and the good guys try to flush it out again and again. Try doing this on about 100,000 servers at the same time and you’ll realize it’s a cat-and-mouse chase.”
Also, Corero’s suggestion to use “flush_all” might subject users to legal repercussions, says John Graham-Cumming, CTO of Cloudflare. “We didn’t use this technique because it’s unethical and possibly illegal,” he tweets.
Servers Vulnerable to Abuse
Shodan, a search engine for devices connected to the internet, reports that as of March 5, more than 105,000 internet-connected servers were running Memcached and answer on TCP or UDP port 11211, meaning their use of the protocol could potentially be abused.
More than 10,000 of those servers appear to be run or hosted by China’s Hangzhou Alibaba Advertising, an advertising technology unit of e-commerce giant Alibaba. By domain, the single greatest number of servers appear to be hosted by Amazon Web Services.
As of Thursday, however, Shodan searches reported that the number of servers listening on port 11211 had decreased from approximately 105,000 to less than 96,000.
On Feb. 27, DDoS defense firm CloudFlare reported that its data centers in New Jersey, Germany and Hong Kong were seeing a disproportionately large number of attacking IP addresses because most of the vulnerable servers are located in major hosting providers.”
ISPs Help Mitigate Threat
Based on scans of the IPv4 space, DDoS defense firm Qrator Labs reports finding fewer vulnerable servers. Whatever the actual quantity is, the good news is that the number of servers running Memcached in a manner that can be exploited by DDoS attackers appears to be diminishing.
“The significant part of this decrease was the result of homework made by several huge ISPs,” Qrator Labs says, noting that by Monday, Alibaba had fixed more than 5,000 servers, hosting giant OVH more than 2,300 servers and China Telecom more than 1,600. “Still, there are more than 8000 services that haven’t been fixed,” Qrator Labs says. “We encourage all ISPs to check their networks and fix services that can be used as amplification for the DDoS attack.”
All ISPs should “port filter (exploitable port filters), rate limit the port 11211 UDP traffic (ingress and egress), and clean up any Memcached exposed to the internet (iptables on Unix works)” for both IPv4 and IPv6, Barry Greene, principal architect at Akamai, says in a blog post cited by US-CERT.
“There is not excuse for ISPs, telcos and other operators for not acting,” he says.
Service providers have also been responding, including telecommunications giant NTT Communications, says Job Snijders, an internet architect at NTT
“NTT too has deployed rate limiters on all external facing interfaces on the GIN [global IP network] backbone – for UDP/11211 traffic – to dampen the negative impact of open Memcached instances on peers and customers,” Snijders writes in a March 1 post to the North American Network Operators Group mailing list.
“The toxic combination of ‘one spoofed packet can yield multiple response packets’ and ‘one small packet can yield a very big response’ makes the Memcached UDP protocol a fine example of double trouble with potential for severe operational impact,” he adds.
What IT Administrators Must Do
Enterprise IT administrators must also act. “If you are an enterprise, ask your upstream ISP if they are deploying exploitable port filters and look through your firewall rules to add them to your network,” Greene says. “Layered security means it is prudent to ask your ISP to do their part while you do your part.”
“There should be no business for these memchached servers to accept connections from the internet. People should be using firewall or routing rules to drop traffic from the internet to these servers, period,” says Juniper’s Hahad.
“It is very easy to mitigate this kind of amplification: just rate limit the particular port used by this service, 11211, which can be done by any decent firewall,” he adds. “Better yet, reconfigure your network to have Memcached only allow connections from the desired servers.”
Ullrich suggests that running Memcached in internet-exposed mode should be a firing offense.
“If you have a Memcached server in your environment that is exposed to the internet, then you should stop scanning for them, and spend your time writing a resume instead,” Ullrich says.
“Either because you do not want to work in an utterly incompetent organization like that, or if you are responsible for the exposed server, then … write a resume for a simpler job” than being a systems administrator, he says.