Fighting DDoS Eyes Highload Lab

    The creator of Highload Lab and QRATOR traffic filtering network Alexander Lyamin talks about trends and trends in DDoS attacks. We took the interview at the beginning of the year, but so far a few DDoS attacks have appeared at the dawn of the Internet. Acquaintance with them for me personally began during the IT Territory project in 2003, when the game was just launched. She had a rather aggressive advertising campaign, in response to which DDoS from competitors immediately flew in. Frankly, I was confused. Most of all due to the fact that the company that provided the hosting services, not just could not, but did not want to fight the attack. Its representatives said that this was not their problem. Literally overnight, we rethought the structure of the application and rebuilt it.





    The resource has risen, but the attack has stopped working at the application level. Attackers transferred the attack to the network level, as a result of which the entire network of the hosting company was disconnected. Then she already came to us with a request to do something. To the question "what to do?" The answer was: "Well, agree somehow, you know who is attacking you." Naturally, we did not know and could not agree. And how to negotiate with terrorists?

    The company, which, it would seem, should take care of customers, did nothing. And not because it is bad and the hosting is of poor quality, but because she could not do anything.

    The most popular DDoS attacks are, of course, attacks organized using botnets. This is an affordable way to make a distributed attack.

    The cost of an attack greatly depends on how it is implemented. The performer may be a student who himself has written something and is ready to work for beer, or maybe an organized group.

    There are types of attacks that can be rumored to cost half a million rubles or more. We distinguish attacks of the basic type: up to five thousand bots, conducting at the application level, one strategy. There is nothing complicated for the performer - I got WMZ, pressed a button, went to drink beer. It costs about $ 30–100 per day. But there are attacks of a different kind, when it is clear that the team is working, and it works 24/7 - on the result. If she is unable to achieve a result, she constantly switches attack modes, changes her strategy, tries to find a weak spot and break through. Of course, this is far from $ 100 per day.

    Russia stands out among other countries with much more sophisticated attacks. Europeans are shocked by how complicated our attacks are. For example, we were recently approached by a company that works in the Russian market, but all of its information structures are located in one of the leading European data centers. When the company contacted us, the data center experienced serious problems and was unavailable. We prepared to receive the company on our network and warmed up the quarantine equipment, expecting something extraordinary, because the data center had “died”! What was our surprise when we saw that, although the attack is carried out at a level above the average, it does not constitute anything special.

    Legislation is extremely weak in matters of holding DDoS accountable,due to which attackers feel the factor with impunity. For a qualified programmer, conducting DDoS attacks on order becomes an absolutely safe and profitable business, the income from which can exceed ten times the current salaries in the market.

    Motives for DDoS are, as a rule, money and just personal hostility. De facto we live in an information society. The speed of dissemination of information affects it directly. By blocking the source of information, you can irreversibly affect society. Accordingly, DDoS is an effective means of blocking any source of information for the necessary time.

    An example is the site Slon.ru.The site worked, everything was loading, but the attack did not subside. Such attacks are called combined attacks. When they came to us, she was led on a network level, to the strip. When the attackers saw that filling the strip does not give any result at all, an application-level attack began. The botnet used to conduct the attack included about 200-270 thousand bots.

    The narrowest areas of Internet business with high competition are most susceptible to DDoS attacks . A good example is the Lineage II pirate clones. Such services are generally a different story, because they are purely commercial. If at two o'clock in the morning someone knocks on you in ICQ and, having made eight spelling mistakes in four sentences, demands (!) That you help him immediately, there is no doubt that it is he! Lineage Pirate Administrator!

    Those sites where it is difficult to expect them are subjected to DDoS attacks. For example, we have such an internal meme - “cedar barrels”. An online store that, you will not believe, sells cedar barrels, has undergone a serious DDoS attack. This is a very narrow, highly competitive type of activity, which, apparently, brings good profit.

    The technical side of DDoS


    Why do we need to classify attacks? To understand, to sort out their mechanism of work and take adequate countermeasures.

    Our workshop colleagues are trying to somehow classify attacks. On one site you can find ICMP spoof, DNS amplification, TCP SYN flood, TCP RST flood - guys list attack techniques. There are a lot of scary letters that for the average user make no sense. Such a classification does not suit us.

    We classify attacks very simply: attacks on applications, attacks on the bandwidth (speed is measured in gigabits / s), attacks on the network infrastructure (speed is measured in packets per second), attacks on the transport layer (TCP / IP stack).

    The pulp itself is the application tier.Why? Because application-level attacks have maximum leverage. Leverage is the ratio of resources needed on the side of attackers to resources needed on the side of the application (on the side of the defenders).

    Take some average online store. You can find a link, a regular link, a certain number of hits per second to which this store will kill completely. There are many such applications, and in order to kill them, sometimes a botnet is not needed - a cell phone is enough even with EDGE, but with GPRS. Four to five requests per second, and the application is thrown into the outflow and cannot exit from there until the server reboots. This is due to the popularity of application-level attacks.

    There is still a transport level- attacks aimed at the TCP / IP stack itself. This type includes attacks like SYN-flood, RST-flood or FIN-way - now a fashionable attack with incorrect connection closure, which, by the way, also exploits a protocol specification vulnerability, not an implementation.

    The most popular methods include DNS amplification.It is enough to find any IDP based network service without a handshake, send a packet of size N with a falsified source and in return receive an N x K. packet. In this case, to implement a distributed attack, you need to have a list of IP addresses that have these services, one very well connected a gigabit server that will issue a set of packets to these “reflectors” with the victim’s IP address. The packet will go to the victim and reduce its throughput to zero and put it out of operation. This is a DNS server - UDP 53, where you can do such a thing as a recursive query on the zone. By itself, it is small, but the answer to it will be long. To further increase the K attack coefficient, it is enough to “feed” these servers with some large fake domain zones. Receiving them recursively, with a fake victim address, can increase K at times. The second option is NTP, a time synchronization protocol that also has vulnerabilities of this kind with poorly configured servers.

    An SYN flood attack appeared simultaneously with the TCP protocol. I met the first mention of her, in my opinion, in 1982. Oddly enough, it is effective to this day. History develops in a spiral. One hundred to five hundred SYN queries per second is, of course, a stage long gone. Currently, with sufficient processing power, you can easily surpass the figure of 10 million packets per second.

    At the time of sending a packet with a connection requestthe partner must generate a sequence-number (and this requires calculations, since it must be a random, crypto-resistant number) and create a specific record in its stack. This requires resources, resources, and once again resources. To send a packet, generating its traffic, requires significantly less resources than actions performed on the server side. We get the shoulder of the attack.

    The problem exists within the protocol itself, in its specification. When the TCP / IP stack was being developed, no one thought that the Internet would grow to such a scale in terms of the number of nodes, reach such speeds and, importantly, pump so much money through itself.

    There are attacks that do not use botnets.A distributed attack can also be carried out using the reflection mechanism. The classics of the genre are DNS amplification attacks with reflection and power increase.

    Attacks on the infrastructure affect everything that lies around the network infrastructure: routing protocols and the equipment itself, if the management modules have an open IP address.

    What are network-level attacks? Just fill in a strip of 56 Gbps - this is called clustery sort. This is the last resort when nothing else helps. Such attacks are very expensive and extremely destructive, not only for the victim itself, but also for everyone who "stands nearby." As a rule, they cannot last longer than two or three days, since they begin to cause problems even to the sources of the attack - the networks from which it is carried out.

    Basic attacks using botnets, which have about 200 bots and can do nothing but get the root, in principle, should not be a problem for a well-written resource.

    About DDoS Protection


    You can protect yourself from any attack. We have no doubt about that.

    Usually, when a DDoS attack starts on a client, the hoster does not find anything smarter than just turning it off, as it is afraid that other clients will fall as well.

    When a client comes to us, we explain that he needs to translate the DNS to our IP address (which we give him). Also, in order to avoid an attack on the direct IP of a client that is already exposed on the network, you must at least change it, and at the maximum - change this IP and additionally hide all IP addresses except for ours using iptables or firewall settings.

    As soon as the DNS is rebuilt, filtering begins.And then the fun part happens - filter training. Usually we set the bar for ourselves: after two hours under attack, the client’s resource should start working. And in general, we stand it.

    The security system of our Qrator service is based on many mathematical constructions. How does Yandex usually answer the question “how is your search arranged?”? Yes, simple! We take the text, tokens, dividers, build indesk, rank. We have about the same thing, only we solve the problem of analyzing and filtering traffic. A lot of people are busy with her decision.

    Behavioral analysis is one of the most effective traffic filtering methods.We consider the site as a transition tree. There are pages in the nodes of the tree, and in the edges we lay the probability of the transition and the delay in the transition. It is based on the simple fact that robots and people see web pages in different ways. When there is enough time for training, people “trample” certain seals in these transitions - paths. All visitors who fall out of them are, with one degree or another, robots. Everything seems to be simple. On the other hand, if you figure out how much memory and computing resources will be needed for processing, you will realize that, probably, with the current computing power it is not so simple.

    If a client comes to us already under attack, training on the attack ... is not impossible, but difficult.Often this requires monitoring by an engineer. That is why we are forced to charge some additional charge for connecting under attack.

    We recommend connecting to our network before a DDoS attack. Of course, we try to minimize the training time under attack. We do not have it for a week, like Cisco Guard (this is our iron competitor, which has been discontinued), but only a few hours.

    Anyone who says that false positive has zero is a quack. False positives (when legitimate visitors are blacklisted), unfortunately, are inevitable. If only because there are proxies, there are NATs, there are just people who do not behave like ordinary users. A classic example is site administrators. The administrator can load the server as 30, 40 and even 100 users.

    We had one complaint with Cisco Guard: when you connect an attacked service to it, then regardless of whether there is an engineer there or not, the first day the service works in such a way that it would be better if it did not work at all. From this it became clear that it is impossible to protect against DDoS attacks at the application level without understanding the semantics of the application protocol. Semantic analysis is required, as is behavioral analysis.

    We clearly understand that there is no “silver bullet”: what will work well in some situations will not work in others. The Qrator classifier is a complex set of algorithms that form a voting system. We are trying to develop the tools and add them and, hopefully, we will find some other effective methods in the near future. Some ideas are already there.

    About one million dollars will cost a piece of hardware from Arbor, capable of cleaning 10 Gbps. Plus a person, plus channel capacities ... At the same time, attacks are observed at speeds above 10 Gbit / s approximately once a month and a half.

    We tend to distinguish two types of bands: active and passive.In the active lane, you can terminate and analyze any TCP connection and make a decision on it. A passive band is a band for which you need to set a bit mask by which traffic will be cut. Thus, something intellectual cannot be cut there. If we talk about the active lane, then almost all of our traffic providers, if necessary, block UDP from a specific address, all ICMP or ICMP according to a specific signature. On this strip, we quietly lived 57 Gbps. We are sure that we can live more. Such attacks do not cause any problems, except for the need to pay for this band, that is, we are talking about a figure of more than 100 Gbps for the passive band. As follows from the situation with DDoS attacks on the Russian market, this is quite enough.

    The advantage of Qrator (as a service) over Arbor purchased is that our solution is not a point solution. The network is built on BGP-AnyCast, we choose to install points exclusively trunk operators. We do not put an end to public exchange simply because this does not guarantee the quality of the service. The network is developing thanks to our own modeling algorithms. We build it so that it is possible to distribute the load on the network elements more or less evenly.

    Inside the point of presence, the system is also scalable. A point is not a single piece of hardware, there are several of them. There is quarantine equipment that some attacks land on.

    We created a model to mathematically calculatehow traffic will be distributed over the Internet when certain BGP announcements appear. This allows us to harmoniously develop and build a network that can really be balanced across nodes.

    We are not tied to one telecom operator and try to distribute risks among all the operators with which we work.

    We tried for a long time to figure out the TCP / IP stack, looked at Free BSD and Linux, and eventually came to the conclusion that we did not like the stack in its current state. We have our own lightweight version of TCP / IP, which behaves very well on current short-lived protocols, fast TCP connections.

    We do not hide the fact that the filtering node is running Linux.Linkus is a container in which the platform is managed and the mathematical transformations necessary for behavioral analysis are performed. A significant part of the TCP stack lives in the TCP card itself, so, in fact, we got such good speed / packet processing indicators. One of our filtering nodes is able to shovel 6 Gbps of traffic.

    You can protect yourself from basic attacks.To do this, you must have dedicated hosting, as well as the ability to compile modules and your version of the web server. A lot of articles have been written on this subject, and I will probably refer you to my article of 2008 (you can find it on the blog on highloadlab.ru). This is one of the first articles in which it is readily stated what and how to do it. I also recommend that you familiarize yourself with the presentation “Practical Guide to Surviving DDoS,” which we showed at Highload ++ in 2009.

    We tried to write articles on Habré and tell at an industry conference how to defend ourselves independently from base level attacks. But, unfortunately, this had no effect.

    About botnets


    DDoS is one of the ways to monetize a botnet, but by no means the most profitable one. There is still spam, fraud, ad clicks and so on.

    I have listed the botnet properties on the slide for one fairly old presentation. When I made that slide, it seemed to me absolutely correct:
    • Greed. A botnet tries to do as much harm as possible per application per unit of time.
    • Damage A botnet is not a browser. This is some kind of HTTP stack built into the worm. As a rule, he does not know how to set the correct headers, does not have a JS engine, or has a limited form.
    • Self preservation. A botnet is a valuable resource, and any actions leading to a reduction in its size bring direct financial losses to the attackers. Botnet tries not to take actions that could unmask it and affect the mother system.
    • Transnationality. Botnets scattered around the "ball".
    • Limb.


    A couple of years ago, the first time we saw a slow botnet that was not greedy ... He made one absolutely legitimate request every five minutes. We were surprised, but at the same time the botnet, numbering 75 thousand bots, still caused problems. Try filtering THIS.

    Now of all the items listed above, there is only one left - the desire of the botnet to self-preserve. Botnets have long been not greedy, not stupid and not flawed. Now we are dealing with full-fledged minimized web browsers with Java scripts, redirects, cookies.

    Distributing commands to members of a 20,000th botnet, given that the bot is the initiator of the connections, is not the most trivial task.As a rule, control panels, to our surprise, are written on the same LAMP Stack (Linux, Apache HTTP Server, MySQL and PHP). Until 2010, the deployment of the five-thousandth botnet in the direction of the victim’s resource took 30–40 minutes.

    In 2010, botnet management began to be organized using P2P. The guys began to simply give out superfast commands: within five to six minutes, a botnet with 10-20 thousand bots can spread the command within itself and deploy to the resource.

    Botnets try to imitate the behavior of users as accurately as possible in order to complicate their detection and filtering, highlighting the botnet's body and blocking it.

    For example, not so long ago there was a surge in the activity of the MinerBot botnet that BitCoin is mining.It comes to the cover pages without a referrer, randomly follows links and really creates problems for solutions like Cisco and Arbor. They are not able to filter MinerBot, because it does not have any of the flaws that these solutions are aimed at detecting.

    Botnets have also ceased to be transnational - downloads are easily sold by region. The first time we saw this in 2009, when a botnet for 1,500 “came” to us, and everything is clean CIS.

    The “falling asleep botnet” is what we call a rather fashionable attack.The botnet discovers that it is all filtered, issues a command and centrally stops the attack. After that, the random member of the botnet sends test requests, waiting for the filter to turn off. As soon as it turns off, the attack resumes in full within three to five minutes. This is dangerous because such an attack can last indefinitely - it does not consume any resources, from the point of view of the botvod.

    Different botnets are great different from each other. The attack technique itself is constantly changing.

    With millionaire botnets, a very interesting situation is observed.In the past few years, the number of those who would like to get their own botnet has increased significantly. A simple experiment: put Windows XP SP1 on an honest IP address. How long will he live before something “lands” on him, even if you don’t open the web browser? Five minutes maximum. There are many teams that struggle to increase the botnet's body, and the supply of vulnerable systems is extremely limited. Accordingly, the number of botnets is growing, but their size is slowly but surely decreasing. Botnets are already starting to intersect, that is, one computer is a member of several botnets at once.

    Botnets for tens of millions of computers are becoming less and less. They are possessed by absolutely Jedi. :)

    We do not have the ability to reverse engineer the botnet code, because we don’t have administrative capabilities to remove his body and, most importantly, we don’t have our own specialists who can do reverse engineering of code oriented to Windows systems.

    About Highload Lab


    The idea to engage in DDoS attacks came up with us at Moscow State University. We looked at how the stability of external resources of government resources and the stability of web applications in Russia as a whole are. It became clear that our services are likely to be in demand. After all, a fallen off online store is the problem of its owner only, but a fallen tax inspection is a problem of the whole country.

    Start research - it was my personal initiative. The university provided the infrastructure, I bought equipment with my own money.

    In 2008, we had an idea.In 2009, a beta version of the product appeared, which we tested in open beta for most of 2010. We hosted any project in distress at our site. It became clear that we are coping well with this task, even with a limited university infrastructure. For example, we helped the newspaper Vedomosti. Was great. :)

    The need to push us towards commercialization:in June 2010, when the maximum capacity of the university network was 10 Gb, an attack of 12.5 Gb fell on us. The attack showed that the filters are coping, and we can easily overcome a more powerful attack, but we need channel capacities. This is a valuable and expensive resource, but I don’t want to lose either ... We had some accumulated funds of our own, for which channel capacities were purchased. Additional equipment was also purchased.

    We were lucky with the launch - we had a wonderful stress test. That is, on September 1, according to the plan, I just set the last entry point, and on September 2, Habrahabr came to us under the attack of 6 Gbps. We got a free stress test.

    Traffic is one of our main expense items. It is spent not just a lot, but a lot.

    The company works in several directions: we develop highly loaded web applications to order and advise on the issues of their creation. The second direction, the most promising for us, the most dynamically developing, is our “boxed” product, the Qrator traffic filtering system. We invest in it practically everything that we earn.

    At the moment, 12 people work in our company. The non-technical staff consists of eight engineers and four other employees. Two of them are freelancers from Moscow. At the beginning of the year, if everything goes well, we want to invite two more engineers to the company. Like Yandex, we are looking for mathematicians who can program, work with data (structured and poorly structured).

    Unfortunately, we don’t do reverse engineering,but we see that each attack has its own signature and logic.

    We have been around for a year and a half. This was not an easy time. At some moments it was very difficult both financially and morally. But during this time we found out what issues arise during the operation of the service, we understood how to form the tariff stack. Since the service is new, no one knows how to sell it. All offers on the market have certain flaws.

    Highload Lab is profitable. This year we made serious technical changes - developed a new version of our specialized network processors - and actively developed partnerships with all interested companies: hosting companies, telecoms.

    One of our goals is to provide protection for small businesses.This is the most unprotected layer. Many companies charge between 50-100 thousand for protection against DDoS attacks, and if a small business pays so much, it will go bankrupt. For small businesses, we have a special tariff - 5000 rubles. But this does not mean that at lower prices we work worse. All our tariffs use the same system, the filtering quality is the same everywhere.

    We are extremely apolitical. During the election, our clients were Elephant, New Times, golos.org, Ekho Moskvy, Novaya Gazeta, St. Petersburg, Forbes, Public Post, Vedomosti ... In general, we took the opposition under our wing. But even with great pleasure we would work with the same CEC. But the CEC did not come to us.

    The only criterion for us is that the resource must comply with all laws.We basically don’t get involved with sites containing pirated content, having a Nazi or pornographic orientation, with pharmaceutical partners and other online dirt.

    We thought that if we could create a system whose construction and operation would be cheaper than carrying out an attack that could kill this system, we would eliminate the economic shoulder of the attack. An attack would be unprofitable. Based on this, we built the ideology of the development of our solution.


    First published in the Hacker magazine from 02/2012.

    Publish to Issuu.com

    Subscribe to Hacker



    Also popular now: