BLACK HAT conference. Survival lessons in a 300 Gbps DDOS attack. Part 1
  • Transfer
My name is Matthew Prins, I am one of the co-founders and CEO of CloudFlare. Do not think that I will be as interesting as the director of the NSA, but at least I will try to be less controversial. In fact, I am a private law professor, so I read the Constitution and can start a long discussion on this topic. Someday we will discuss this in great detail, hopefully I can tell you the story of the lawsuits that we are currently conducting against the United States government to dispute certain things that are happening, since this is perfect rubbish.

I will tell you a less interesting, but dramatic story that took place from March 18 to March 25, when one of our clients underwent the strongest DDoS attack. I will not talk much about CloudFlare, if you want to learn about our service, just go to the site. But when our client came under attack, it was an event from the line of those journalists like to write about, and this attack was covered by the New York Times for 4 days in a row. It was a really powerful attack, so I want to talk about it.

So, there is a non-profit organization called Spamhaus, which tracks spammers and their activities. I know these guys for a long time, since 2006-2007, when an alleged spammer from Illinois sued them. Then the founder of this organization, Steve Linford, called me and asked if I could speak for them as a free attorney. I replied that I used to be a private lawyer, which means that I taught students in the class, but I never spoke in court. I very quickly found a real lawyer for them, who soon helped them to "resolve" the situation, so I know that Spamhaus is a very decent organization, they act quite openly.

Our clients are many organizations, but they don’t like to talk openly about their problems, but the guys from Spamhaus are not so closed, and when they were attacked, they called us and said: “Hey, we can tell you our story”! So this is a story about the attack on Spamhaus and about what happened behind the scenes when they were "brought down".

I broke this story into 3 parts. The first is the nature of the attack, what it is and who is behind it. If you want to know more about this, then come to DefCon, where I am going to speak next week.

The second part is what needs to be done to stop the attack, how to create a network that is really capable of it. This is not a story about great intelligence or unbelievable software, it’s really the story of designing a smart network.

The last part is what you can do on your own networks to stop this type of attack. I promise that the outcome of the speech will not be campaigning to use the CloudFlare service as the most protected, instead I will tell you about the advanced things, having done that, you will become less vulnerable to such DDoS attacks.

So, on Monday, March 18, we received a call from one of the Spamhaus volunteers, who informed us about the attack. This is how their website looked like at that time - the attackers brought it to the offline state, effectively disconnecting the Internet.

Later, the media wrote that the attack was made by someone Fenn, who lives in a bunker somewhere in the Netherlands. This guy was not the real initiator of this action, he was probably just the one who was willing to give an interview to the New York Times. Rather, the real mastermind of the attack is the 15-year-old teenager from London, who is now under arrest, but the press does not make a fuss about it.

In an article dated 23.03.2013, the New York Times newspaper wrote that the attack was carried out by the Danish company Cyberbunker in response to the fact that Spamhaus was blacklisted for spreading spam. The headquarters of Cyberbunker was indeed located in an abandoned NATO bunker (note the translator).

Most likely, the motive of the action was the fact that Spamhaus declared someone a spammer. This organization does a lot of good things, but it happens that they pinch someone’s tail, and in this case, the offended side responds, in this case it organizes the attack. Spamhaus is technically savvy enough to repel many different types of attacks, but in this case, a massive group attack hit their DNS infrastructure and simply knocked the site out of the network. It could happen to any organization, and they wanted to stop it. They knew that we were helping to prevent such attacks, so they turned to us.

So, from March 18 to March 21, what we call the "tantalizing attacks" took place. Such attacks are our bread, weekly we have several such attacks. So that you have an idea of ​​their scale, I will say that last week we recorded 162 attacks on our clients, and we rather successfully stopped them. These DoS attacks occur at an intensity of 10 to 80 Gigabits per second. For some organizations, such as financial ones, attacks that occur with such intensity can be dangerous. So, last year, some financial institutions suffered from DoS attacks of 60-80 Gbit / s, but I repeat - these are fairly common attacks, familiar to us, and there is nothing in them that deserves mention at the BlackHat conference.

Another thing to note is that the DoS attacks are not all the same, the size of the attack, that is, the number of packets attacking the site, is important. What we will talk about today refers to attacks of level 3, that is, attacks against the network layer of the OSI model. This is pure volume attack. But it is not the worst, much worse than the attack of the 7th, application level, which is not so voluminous, but it creates much more problems for specific applications. But today I am not going to discuss attacks that require special software.

The same applies to Level 4 DoS attacks, when an attacker uses a small amount of Syn-packets, causing significant damage to the victim, because his operating system is not able to cope with these requests.

In our case, the attack is based solely on a large volume, which tries to destroy you at level 3, knock you out of the network and overload the port with traffic. On Wednesday, March 20, one of the ports of our network, or rather, a collection of ports, on one of which the Spamhaus site sat, was attacked.

This slide shows the proxy cache, the blue line is traffic coming from our network, and the green area is incoming traffic. The high peak at the end of the chart is an attack that fell entirely on Spamhaus and brought down our network. Our network can easily withstand attacks of 75 Gbit / s intensity, because it is designed to “digest” such a volume of traffic.

However, during the attack on March 20, the “ceiling” of DoS intensity was reached - 100 Gbit / s. This looked like an exceptional case, because if you buy or create a basic router, then contact someone like Cisco or Juniper, and the coolest and largest “box” you can buy from them has a port with a capacity of 100 Gig. Theoretically, you can use several 100-gigabit ports and pass them through one interface, but the ceiling of DoS attacks usually does not exceed the specified amount.

Soon after, Tom phoned me, one of our network engineers, and said that the intensity of the attack had increased. She can push herself beyond the boundary values, so I asked how much she grew up, thinking that we are talking about a slight excess. The next slide shows the intensity of the new attacks that took place on March 24-25.

The graph shows the universal coordinated time of UTC, I live in San Francisco, we had another 23 at this time, so you can imagine what my dinner was on March 23. They called me and said that the attack exceeded the ceiling of 100 Gigabits, but we did not think that very soon we would be able to see something like that! Suddenly, 4 calls were received from the provider of one of our incoming networks and network engineers that we now have traffic with an intensity of more than 300 Gbit / s, which reaches the limit of the capacity of our network.

The graph shows that throughout March 24-25, attack rates dropped to 90 Gbit / s and increased to 300 Gbit / s, causing a denial of service to the incoming traffic of one of our providers, which has completely exhausted network bandwidth.

The question is how? How did a hacker manage to do this if it is known that the ceiling of the equipment is 100 gigabits? How did this guy from London, this 15-year-old, manage to organize such an unprecedentedly intense attack?

Many people, including the media, expressed the wrong opinion that a botnet is needed for such an attack. But in this particular case, there was no real traffic from the botnet network. Other people said that it was Anonymous, because there are many people in this community who are coordinated and can work together to launch attacks of a certain kind. No, it is not. The attack was carried out by 3, possibly 4 people, one of whom was a real technical inspiration, but very limited in terms of resources and performers.

Many people felt that outstanding technical skills were required to carry out such an attack. I say that I saw the code that provoked this attack, which is about 50 lines in C, not including the headers that are usually included in such an exploit, so this is quite a bit of information.

In just a second, I will show you how you can launch such an attack. What is really needed for this is the list of open clients DNS, or DNS resolvers, later I will tell you what it is. The second thing you need is a few servers, or just low-powered computers running on networks that allow you to replace the original IP.

These are the basic components for attacking, and if you have these two things, as it turns out, even in very small quantities, then you can launch a very, very large attack. So far, nothing that weakens such attacks has changed, and this is a bit creepy.

Let's talk about open DNS resolvers. DNS is something like small blank pages for the Internet, it is a caching system that creates the ability to respond to requests that were sent to the server.

An improperly configured DNS resolver, also called an open DNS resolver, answers questions no matter who asks it. This “slutty” DNS resolver answers everyone, no matter who asks. It works without any restrictions. It has no restrictions on the number of requests, or on who to receive requests from.

You may ask: “What about Google? After all, he uses an open DNS resolver! ” However, organizations such as Google are doing a lot to limit the number of requests addressed to resolvers, in order to make it difficult to launch attacks, they limit the size of the answers that are provided on requests. Just last week, the guys from Bind announced a new extension that allows them to set a speed limit for requests, this is exactly what we lacked for a long time. Because what scares you most is that authoritative, open DNS servers are increasingly being used for attacks.

When carrying out any attack, the hacker seeks to strengthen it in any way possible, for example, using a botnet. If I have very few of my own resources, I want to use them to create additional resources. At the BlackHat conference, it was said that DDoS is a perfectly asymmetric attack, because one little person can create huge problems, and open DNS resolvers clearly demonstrate this.

This slide shows the key command to launch an attack through a Windows terminal, or console.

This command was launched at 2 am from an open DNS resolver, which worked on the PCCW network. If you send this request, the server will respond to you.

digANY is just a DNS query that initiates a search for any domain names NS, isc. org means that they wanted to show you every possible type of DNS record, so their DNS file is huge, followed by the resolver IP address and the flag edns = 0, which says: “Give me everything, including dns.sec and many others flag types, which you may not get by default. The notcp flag is an important element, because it orders the resolver not to use TCP for data exchange, but to send them all over UDP, and bufsize sets the buffer size to 4096, that is, to make the buffer as large as possible to send very “fat” packets.

So, you run this cute little query of 64 bytes in size and get back what is now running on the screen. That is, on a 64 byte request, you get a response of 3363 bytes in size.

So, if you manage to find an open DNS resolver, you send it 64 bytes and receive a response of more than 3300 bytes in size. That's what I mean when I say that you can use tiny resources to create a big problem. In this case, we get an attack gain of about x50.

However, the gain does not work by default, and if I send a small request to get a big answer, I just send a DDoS attack on myself. Therefore, the second mandatory component for such an attack is a network that allows the source IP address to be spoofed.

That is, I try to pretend that the DNS request was sent from an IP address from which it was not sent. I send a message to the DNS resolver, and since UDP is a “burn and forget” protocol without a handshake, after receiving the request, it simply sends the answer to someone it considers as the sender.

Well-established networks do not allow this, they do not allow packets to come from IP addresses that do not belong to this network — these are BCP38 recommendations. The majority of Internet providers follow these guidelines when creating networks, however there are networks that ignore this rule.

So, if we have two necessary ingredients, then since UDP does not use a handshake, you can easily change the IP address of the source of the request. For the attack on Spamhaus, the specific DNS shown on the slide was used, and the confidence that the connection is made via UDP, which does not require any handshake. Then, the responses of all of these open DNS resolvers located around the world are sent back to yours or any other target network.

The secondary effects of these DDoS attacks affect customer support, because hundreds of thousands of open DNS servers are involved in this attack. Their users track the path to an attacker address, from which tons of packets emanate from their infrastructure, and begins to flood you with angry messages like “you don’t know what you are doing, you don’t know how to run a normal network, your computers hacked, how dare you attack me! ”and the like. And you have to be very polite to explain to them that in fact they are attacking you, and they themselves suffer because they have not correctly configured their DNS resolver, the answers to the queries come back, and all this ultimately affects you. themselves.

Actually, this is an opportunity to teach people to “clean” their networks, and this is the positive aspect of such attacks, because people do not realize that this happens all the time, due to the incorrect configuration of open DNS servers, only on a small scale. And only when the intensity of attacks increases significantly, they begin to pay attention to it.

I think all of you are familiar with such old types of attacks, like Smurf, where you use the transmission of ICMP broadcast requests to routers via routers, and use source spoofing, causing computers to respond to the victim of your spoof by hammering communication channels.

On the positive side, the routing industry has learned to deal very effectively with this attack. They are still found, but it is quite difficult to launch the Smurf attack. However, in general, attacks by reflecting DNS responses are very easy to organize.

How common are the components of such attacks? The next slide shows the Open Resolver Project, this is quite a revolutionary product. Many of those present here argued for a long time whether to publish on the network a list of all open DNS resolvers running on the Internet. The fact is that this list can be used both by the “good guys” so that they can point clients to incorrectly configured networks, or by the “bad guys” who can use it to organize DoS attacks.

It is not by chance that this project was launched on Monday, which followed the weekend, during which we endured all this hell. Thus, the largest Internet providers of the first level were able to make sure that the “bad guys” have this list for a long time and therefore it is necessary to start cleaning it, getting rid of open servers. Thanks to this site, you can enter a range of your IP addresses and check whether there are open DNS resolvers on your network. Thus, if the PCCW did a similar check, they could clear the network before they had problems. We attracted the media to cover the role of this project, so that all providers know about this list, because the number of open DNS resolvers used on the Internet is 28 million. I would be happy to inform you that today this number is 30, 40 or 50 million, however, unfortunately,

The next thing I want to mention is the completely abnormal percentage of networks that allow spoofing. The slide shows the MIT project that allows you to track the spoofing networks. It was difficult to check them remotely, but tests show that almost 25% of the Internet allows spoofing.

In 2013, there can be no excuse for not using BCP 38 for border routers. We must not allow packets that are not born on your network and that come from unknown IP addresses to increase the load on the routers. Any router released after 2002 makes it very easy to enable such a function, so do not use it - just laziness!

The good news is that mostly large botnet networks do not support this kind of spoofing. Global networks are mostly closed for spoofing, so a catastrophic scenario is possible if hackers succeed in spoofing, for example, China Telecom's Chinese network. In this case, a botnet could be used to generate traffic to DNS servers to carry out a potential huge attack. It is probably easy for those present to understand how difficult it is to get rid of all open resolvers. But you can implement in your own networks the ability to ignore other people's packages in order not to undergo such an attack.

Let's look at two specific incidents that occurred during the attack on Spamhaus. The first was on March 24 and consisted in an attack intensity of about 309 Gbit / s for 28 minutes. The panic began with a call from Tom, who said that we had an attack of more than 300 Gbit / s. I asked if our network works, and Tom said that while we hold on, we can provide Spamhous and other customers. And although there are small packet losses and delays in the transmission of traffic, but in general the network keeps. Then I said, why are you so scared? They answered - because the attackers use only 0.1% of open resolvers, which exist on the global Internet! But if they can use at least 1%, we will have not a 300-gigabit, but a 3-terabit attack. If they use 10% open DNS resolvers, we will get a 30-terabit attack. I think, that today we could withstand up to 3 terabits of traffic, but in March we did not have such an opportunity, we could not beat a 30-terabit attack, and I do not know anyone who could withstand it. This was a very serious challenge to network performance.

When we began to figure out where the attack came from and looked at a bunch of first-level network providers, we found only 3 networks that allowed the spoofing of addresses, and perhaps served them from 5 to 7 compromised servers. One was in San Jose, the other was in Minnesota and the third in Germany. Only 3 networks were involved, and these modest resources led to a relative large attack.

The next peak came on March 25, when traffic was recorded at a speed of 287 Gbit / s for 72 minutes. Practically the same number of open resolvers was used - 31154, and the same three spoofing networks. I repeat once again - the attacker used very limited resources.

I note that we had only 1 attacker with one laptop, we have a screenshot, which I will show later. He controlled from 5 to 7 compromised servers, 3 networks that allow the substitution of the original IP addresses that generated traffic of about 9 Gbit / s intensity, aimed at 0.1% of open DNS resolvers. And all this led to the emergence of 300 Gbit / s DDoS traffic.

Many people believe that it was the biggest attack in the history of networks, I can not say whether this is true, but it really was the biggest of all that we have seen.

This is a story about a great organizer who managed to do something that they had never even guessed before. We all knew the problem of open resolvers, it was not some clever new burglary, damage to network protocols, packet fragmentation attacks, and so on, it looked as if a caveman was simply hammering a huge club on your head.

Again, what happened was an incredibly effective attack and there are very few networks on Earth that can withstand it. Again, we are very proud of the fact that we did then and what we continue to do every day, and that we continue to invest in our network to make it bigger.

I will tell you a little about what we have done in order to be able to resist the attack, what we have taken to stop it. Some of our measures apply generally, some are unique. In this slide, you can see that we provide services worldwide, with 23 points of connection to the CloudFlare service, and by the end of the year their number will increase to 50. So today we have a developed X-class network.

We initially thought about the feasibility of our network, but what was laid in it at the beginning, today turned into a very powerful tool. We use a protocol called Anycast to send BGP packets through a variety of DNS providers that also use Anycast. In fact, we use it for HTTP traffic, as the most stable. I will say that this is a big headache - to ensure the normal operation of a network like ours.

If you are not familiar with the Anycast protocol, this is what it is. In the usual network scheme, there is one source and one destination, and in Anycast, one source is associated with several destinations, and the shortest path is chosen for delivering the packets to the addressee. Thus, if we have 23 data centers around the world, they all have the same IP address, for example, Spamhous address. Therefore, when we deal with an attack of 300 Gbit / s intensity, it is distributed among all data centers in such a way that about 13 Gbit / s traffic falls on each data center.

The problem is that the attacker can also do mathematical calculations and very quickly redirect the attack right to the edge of our network. This is a slightly foolish undertaking, since as a result an attacker may have the opposite effect to gain. That is, instead of increasing the attack by increasing its surface, he is forced to reduce this surface and reduce the intensity of the attack.

29:00 min.

Conference BLACK HAT. Survival lessons in a 300 Gbps DDOS attack. Part 2

Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending to friends, 30% discount for Habr users on a unique analogue of the entry-level servers that we invented for you: The whole truth about VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps from $ 20 or how to share the server? (Options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).

VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps until spring for free if you pay for a period of six months, you can order here .

Dell R730xd 2 times cheaper? Only we have 2 x Intel Dodeca-Core Xeon E5-2650v4 128GB DDR4 6x480GB SSD 1Gbps 100 TV from $ 249in the Netherlands and the USA! Read about How to build an infrastructure building. class c using servers Dell R730xd E5-2650 v4 worth 9000 euros for a penny?

Also popular now: