Four facts about memcached amplification

This is a translation of the original publication by Artem ximaera Gavrichenkov “Understanding the facts of memcached amplification attacks”, published on the APNIC blog (Asia Pacific Network Information Center).

The week from February 25 to March 3 was high in terms of memcached-amplified DDoS attacks in all corners of the world, that is, on the Internet.

Nevertheless, let us once again recall all the facts that we know about amplified attacks.

Fact number one: Amplifiers were, are and will be

NTP (Network Time Protocol) was the first protocol maliciously used as an amplifier of DDoS attacks in 2013. Thousands and hundreds of thousands of NTP servers at that time were deployed throughout the network, so the use of this amplification vector was quite beneficial for attackers. And NTP gave such an opportunity that it resulted in a wave of amplified NTP DDoS attacks. At the beginning of 2014, for a while, NTP as the main amplifier became even more popular than the DNS (Domain Name System) protocol.

In 2015, we saw a massive rise in botnets. They used UDP (User Datagram Protocol) -based amplifiers, setting a record for the band of DDoS attacks observed at that time. Different botnets used different techniques, but the combination of thousands of compromised devices and NTP or DNS amplification made such attacks extremely dangerous, as the denial of service by Dyn and OVH showed in 2016. Joint opposition to the command and control centers of botnets (C&C) made it possible to avoid potential capture botnet control over the entire network.

The same year 2016 clearly demonstrated that NTP and DNS are not the only protocols used as amplifiers. Portmap, SNMP, SSDP, Chargen, MSSQL, CLDAP and some other network protocols provide excellent opportunities for enhancing DDoS attacks. Despite this, the very next 2017 brought a wide variety of vulnerabilities that could exploit the dark sides of other protocols, because attackers with knowledge do not go on vacation and try to look for new attack vectors that have not been studied before.

Fact number two: Amplification has not changed and has not evolved - just found new vulnerabilities

Moreover, it is vulnerabilities that are the reason why we witnessed the events of late February and early March 2018. After a group of Chinese researchers from 0Kee described the amplification path through memcached in 2017, it took the attackers only a couple of months to set a world record for an amplified DDoS attack using an unprotected memcached server. Being free, free and present in almost every Linux distribution, memcached is a powerful destruction tool, providing a hypothetically infinite amplification factor (according to our data, the amplification coefficient is more than 10,000x, according to Akamai - more than 50,000x).

Link to the original presentation 0Kee.

When 10 years ago a wiretap of UDP traffic was added to the basic memcached configuration, the birth of a new amplifier was simply obvious. Just ten years ago, amplifiers as such did not bother almost anyone. Only now, in the context of the growth of packet and packet processing rates that has been going on for two decades, has this become a problem.

The general concept of reflection and amplification has not changed at all. First, illegitimate requests are sent to the vulnerable (open to the world on port 11211) memcached server (this technique is traditionally called IP-spoofing) After that, the UDP server prepares the response and, with the help of the attacker, sends thousands of responses to the target, filling the target host with this completely legitimate from the point of view of memcached responses. The enormous values ​​of PPS and BPS cannot be bypassed by any general methods and means, since such a volume of traffic quickly makes routing devices at network boundaries inaccessible due to exhaustion of the resource.

Fact number three: When something works, it works; but sometimes against you

So, a new world record has already been set in the DDoS attack band: Arbor Networks showed 1.7 Tbps for one of its clients, while GitHub under the protection of Akamai survived 1.3 Tbps. The attention of many media was riveted to the attack on GitHub, as it is a large and popular resource, the normal operation of which relies on many services and products around the world. Qrator Labs client - Qiwi payment system, successfully neutralized an attack of 480 Gbps amplified memcached UDP traffic.

Shodan statistics still show a large number of active memcached amplifiers open to cybercriminals from anywhere in the world:


Shodan statistics on port 11211

As can be seen from this screenshot, mainland China and the United States are the two main sources of memcached amplifiers. Qrator.Radar also collected some statistics on the number of accessible and vulnerable memcached servers:


Qrator.Radar statistics on the total number of vulnerable memcached servers The

number of such servers drops very quickly. Why? No one wants to kill or be killed. Operators (classic telecom providers and transit operators) worry about their networks and that they can be used as a tool to reinforce what can easily be called "illegal."

In order to make this information even more obvious and understandable, we have collected statistics on the number of amplifiers in the customer cone of individual autonomous systems, here is an example at the top of this list:



DDoSmon also provides the ability to get quite detailed statistics on the number of targets for memcached-amplified DDoS attacks, where relevant trends are also visible: https://ddosmon.net/memcached_amplification_attack

Fact number four: everything that can be fixed must be fixed

Memcached users should disable UDP support if they do not use UDP for data transfer. Also memcached should listen only to localhost - not all available interfaces, as it happens in some default installations.

Any player larger than a single company that is responsible only for itself or one person responsible for a particular resource must be 100% sure that no memcached servers are visible from the rest of the network and are closed by a firewall.

Be sure to pay attention to the recommendations of AUSNOG (Australian Network Operators' Group) for the implementation of Explotable Port Filters.

And while the number of vulnerable servers on the network is reduced, attackers continue to search for new and stronger ways to exploit vulnerabilities and, in general, currently succeed in this search. System administrators, who are supposedly responsible for memcached servers, behave recklessly or even ignorantly, so there is no escape from trying to close the remaining caches that cause problems from a long tail. That is why it is so important to try to deal with the problem of operator-level memcached amplification.

At the moment, the best way to do this is to apply rate limits on all ports looking outward and control UDP traffic on port 11211 until it can be controlled. This protects your infrastructure, your customers, and any casual victim.

It contains several recommendations at once: http://www.senki.org/memcached-on-port-11211-udp-tcp-being-exploited/

NTT recommends adding memcached UDP / 11211 to the same “exploitable ports” list as NTP, CHARGEN, and SSDP. The following is an example configuration for IOS XR that allows you to set the rate-limit on amplification-sensitive UDP ports up to 1% of capacity. Qrator Labs considers this method of processing memcached traffic to be extremely useful and effective, recommending it for deployment:

ipv4 access-list exploitable-ports
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
ipv6 access-list exploitable-ports-v6
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
class-map match-any exploitable-ports
match access-group ipv4 exploitable-ports
match access-group ipv6 exploitable-ports-v6
end-class-map
!
policy-map ntt-external-in
class exploitable-ports
police rate percent 1
conform-action transmit
exceed-action drop
!
set precedence 0
set mpls experimental topmost 0
!
class class-default
set mpls experimental imposition 0
set precedence 0
!
end-policy-map
!
interface Bundle-Ether19
description Customer: the best customer
service-policy input ntt-external-in
ipv4 address xxx/x
ipv6 address yyy/y
...
!
interface Bundle-Ether20
service-policy input ntt-external-in
...
... etc ...