May 16, 2011 at 04:06"Caravan", Serverclub and QRATOR. History, experience, pitfalls
Greetings.
After the events of March 31, we finally decided to move from the Caravan.
For those who do not know or have forgotten, one of the most famous Russian data centers on March 31 several times in a row de-energized their “patients”. Among them were Habr, Freelance, Leprosy and others, among which we were. And since we stored DNS records on our server, believing that the server is in safe hands, we were left without projects for almost a day.
Sign at the entrance to the DC. For me, one of the founders of 2Comrade, it is positive. But for dozens of customers who visited DC on the night of March 31, it evokes not very pleasant memories.
This article does not have “paid advertising,” I call things, companies, and people by their proper names.
Thanks to the habrasociety, many of the tips on “where to go” were very “on hand”. Once again I caught myself thinking that it is easier and cheaper to rent quality servers abroad than to buy and host servers in Russia. Surprisingly, for the rental of a 1U server with 8 gigabytes of RAM, Xeons and a gigabit channel in the Netherlands, I give $ 152 (the CBR rate is now 27.9, i.e. it's 4436 rubles) per month. Whereas for hosting my server and renting a 100mb port in Caravan for almost 5300 rubles.
After much deliberation, taking into account the experience of the user and the opinions of the siadmin, I was more inclined to hetzner.de . Until I received an interesting personal message from the user:
“Greetings, and move to our Netherlands, the ping will be excellent and we will give you a discount and the admins will help you move.serverclub.com or here habrahabr.ru/company/serverclub/blog/115169/ . "
By the way, a digression. We bought a caravan protection service from DDoS half a year. Almost, it's + 30k per month. The manager, Sergey Korsunov, assured that this service involves connecting a tsiska, which guarantees protection against an attack of 1 Gigabit. After half a year of renting this thing, an attack of 100 + megabits came to us (approximately 150 megabits ahead of the admin cycles). We lay down. To the question “why?”, The caravan at first could not answer, then answered “the tsisk is not configured, the attack is not blocked”, then answered “the fact is that the attack is more than 100 megabits, more than your channel”, but nevertheless settled on the version “ the tsiska was not configured correctly. " To my attempts to get notes of conversations with the manager, the caravan answered "we are not recording."
In the caravan we kept 2 of our servers. One was under the base, the other under front + back. In serverlab I took 2 servers under front + back and one under the base. The fronts got the simplest configurations, the base received a cool server. For the money it turned out a little more than in the Caravan, but I was very attracted to gigabits, Dell branded servers and it was just interesting. It is worth noting that I am not a fan of brands, for me the cheaper the better. I really don’t understand why take a brand server there and overpay for a brand, when you can assemble non-brand components and get the same result almost 2 times cheaper. Yes, you need to know the intricacies of the components, who and with whom is better plowing, but it does not cost x2 cost. But for such a price as abroad ... it's like buying a liter of Jameson in Duty Free for 700 rubles, while in Russia the cheapest I found for 1000 with a penny (wholesale supplier). A nice bonus for it was the built-in KVM module. Those. the server data at the server club were equipped with a KVM module, which meant that I could always and at any time connect to KVM, completely free. At one time I remember this appeared on SUN servers. It is worth noting that on March 31, in the caravan, when there was a full star, the queue at KVM reached several hours, while KVM remained a paid service. We could afford to come to the DC and figure it out on the spot with the server, which in fact simply destroyed part of the FS and expected fsck to start manually due to an emergency power outage. But what about, for example, regional customers? wait a few hours for KVM, pay for it and either run fsck manually or say goodbye to your FS? Caravan - you freaks.
In general, I was satisfied with the choice and instructed the administrator to transfer everything to Serverlab. The transfer was not without complications. Serverlab easily and quickly allocated servers to the fronts, but pulled with a powerful server under the base. At first he was not in the DC, then it turns out he was set up wrong. The guys immediately identified the “analogue” of a slightly less weak configuration, they say first here, but as soon as we set it up, you will move. Hemorrhoids, but in general it’s pleasant ... worrying ...
I was worried about ping for a long time, but in the end I saw that it was not so much different from Russia to stay in the country. For example, if in a caravan the average response time was 80ms, in the Server Club from American servers the response time was 200ms. A little more than 2 times, but for the user is completely invisible.
A few days later we completely moved and healed in Holland on high-quality gigabit American channels. The guys from Serverlab crossed themselves and continued their work. I am not the easiest client and have actively used an “individual approach” to clients. A low bow to the managers, they got to the fullest ... But we moved. Fortunately there was no limit. Suddenly ... competitors woke up. We are a rather unique service in RuNet. We have only 3-4 competitors that appeared later than us. We provide a taxi exchange service for taxi companies in Moscow, St. Petersburg and other regions and are the largest taxi exchange service. On the day that we did not work and the next day of unstable work, customers partially migrated to competitors. We are not offended, this is normal. Say, if mamba.ru closes, everyone will switch to loveplanet.ru and this is normal.
DDoS attacks went on us ...
serverclub stat recorded 1 gigabit of incoming traffic. I couldn’t physically fix more statistics. The guys from the serverclub said that at that moment the total input to the server increased by 4-5 gigabits per second.
I have long thought to give protection from DDoS to professionals, but I wanted to defeat DDoS myself. By tuning the code and nginx, it generally worked out until the attack grew wider than the channel.
It was 10 p.m. And the qrator.net addresswas typed not the first time. The guys said that they work seven days a week, around the clock) it was a good way to check.
Earlier, I heard about the curator, more precisely about the Moscow State University, which for free protect the server from DDoS by studying attacks. Now they have grown into a paid curator.net service, which is generally logical. My first question, when I got through to them, was “Why are you bringing us?”. I really did not imagine how much an attack in 4 gigabytes and, most importantly, who has such resources can cost. Manager question plunged into a stupor. After half an hour, we had already agreed on everything and after about an hour stood up under their protection. The attacks did not stop for another two weeks, but they were no longer the main problem and went by the wayside.
The thing is that we turned out to be terrible clients not only for the server club, but also for the curator. Imagine ... One service, useful attendance of 500 (!!!) unique users per day. Not 500k, but 500. Less than classmates.km.ru 3 servers and a channel in gigabit. And just do not need to tell me about optimizing the code, the base, etc. I kept traffic at 1 million hits on nnm.ru on a relatively weak server at the time. And these 500 hosts had to be filtered from hundreds of thousands of bots.
The curator could not cope with his task, he kept blocking useful clients. Is it worth repeating that having only 500 useful hosts, everyone was important to us. The co-owners of the resource insisted on abandoning the “Curator”, but nevertheless the joint actions of our developers and researchers of the curator yielded results. At the moment (about a month), all our customers get service without any problems, while all bots are in the forest. The curator sometimes fixes attacks, but they do not affect the resource’s performance, which is insanely pleasant.
By the way, who did not use the services of the “Curator”, I will do a little “review”, at least of what interested me.
Unlimited to gigabits costs 17k rubles per month. In this case, under gigabit, I will introduce “useful” traffic. The site also indicates that an additional dedicated IP costs almost 2 times more expensive, it is not. In fact, an additional IP costs 5k rubles. Well, and accordingly, if you have several projects on one server, you do not need to start up for each IP. The curator’s IP is attached to your IP or IP array (2.5.10, not important). If you have 100 projects on one server, all 100 will be protected for 17k rubles per month.
However, a connection during a DDoS attack costs + 6k rubles. The guys explain this by the fact that they also pay for traffic, and until their classifiers learn to ignore left traffic, they pay for it. And, accordingly, if it exists initially, you cannot learn to filter it every second. There are no complaints.
Upset only paid APIs. The curator has an API, you can manage whitelists and blacklists. However, this pleasure costs money. 5k connection and 3k per month. When the curator blocked useful users, we were determined to take the API, but common sense suggested that paying for anti-DDoS + to prevent users from being blocked is not correct. As a result, he gave a command to developers at each POST authorization to add IP to the white list through a personal account. In your personal account, you can add IP to white lists.
Perhaps it is worth noting that the curator copes with standard projects with a bang. We supported and support the site of the film Burnt by the Sun 2: Citadel"And from the first days they hung them under the protection of the curator. We survived several serious attacks and not a single complaint from users about the inaccessibility of the project.
Today, May 16, a month and a half has passed since the moment we thought seriously and moved from the Russian DC to a foreign one. During this time we have experienced a lot. Anger and threats of “Caravan”, pleasant surprise at the prices of foreign DCs, DDoS attacks of 4 gigabits and experience with the “Curator”.
In fact, a pleasant experience in a short time. But seriously, it’s a shame when the famous Russian DC easily turns off the power without warning and considers this the norm. When the tsiska lease ends with the excuse “oh, we forgot to set it up”, while our service “lies”. When for a project with 500 hosts in Russia you need to think about a gigabit port and an AntiDDoS attack, because "Competitors are not asleep." And that you have to buy a voice recorder in case “we draw your attention to improve the quality of service, all conversations with the operator are recorded” is a lie.
After the events of March 31, we finally decided to move from the Caravan.
For those who do not know or have forgotten, one of the most famous Russian data centers on March 31 several times in a row de-energized their “patients”. Among them were Habr, Freelance, Leprosy and others, among which we were. And since we stored DNS records on our server, believing that the server is in safe hands, we were left without projects for almost a day.
Sign at the entrance to the DC. For me, one of the founders of 2Comrade, it is positive. But for dozens of customers who visited DC on the night of March 31, it evokes not very pleasant memories.
This article does not have “paid advertising,” I call things, companies, and people by their proper names.
Thanks to the habrasociety, many of the tips on “where to go” were very “on hand”. Once again I caught myself thinking that it is easier and cheaper to rent quality servers abroad than to buy and host servers in Russia. Surprisingly, for the rental of a 1U server with 8 gigabytes of RAM, Xeons and a gigabit channel in the Netherlands, I give $ 152 (the CBR rate is now 27.9, i.e. it's 4436 rubles) per month. Whereas for hosting my server and renting a 100mb port in Caravan for almost 5300 rubles.
After much deliberation, taking into account the experience of the user and the opinions of the siadmin, I was more inclined to hetzner.de . Until I received an interesting personal message from the user:
“Greetings, and move to our Netherlands, the ping will be excellent and we will give you a discount and the admins will help you move.serverclub.com or here habrahabr.ru/company/serverclub/blog/115169/ . "
Serverclub.com
I looked, got acquainted with the services, it became interesting. Most of all, of course, I was pleased with the high-quality 1Gb channel and the “individual approach”. The administrator was against it, prepared that the hatzner is cheaper, and the channels there, although 100Mb, are of high quality. Gigabit interested me rather than because it was necessary, but it was simply interesting. Besides the fact that DDoS attacks often came to us, it was just interesting to see, “how is it?”.By the way, a digression. We bought a caravan protection service from DDoS half a year. Almost, it's + 30k per month. The manager, Sergey Korsunov, assured that this service involves connecting a tsiska, which guarantees protection against an attack of 1 Gigabit. After half a year of renting this thing, an attack of 100 + megabits came to us (approximately 150 megabits ahead of the admin cycles). We lay down. To the question “why?”, The caravan at first could not answer, then answered “the tsisk is not configured, the attack is not blocked”, then answered “the fact is that the attack is more than 100 megabits, more than your channel”, but nevertheless settled on the version “ the tsiska was not configured correctly. " To my attempts to get notes of conversations with the manager, the caravan answered "we are not recording."
In the caravan we kept 2 of our servers. One was under the base, the other under front + back. In serverlab I took 2 servers under front + back and one under the base. The fronts got the simplest configurations, the base received a cool server. For the money it turned out a little more than in the Caravan, but I was very attracted to gigabits, Dell branded servers and it was just interesting. It is worth noting that I am not a fan of brands, for me the cheaper the better. I really don’t understand why take a brand server there and overpay for a brand, when you can assemble non-brand components and get the same result almost 2 times cheaper. Yes, you need to know the intricacies of the components, who and with whom is better plowing, but it does not cost x2 cost. But for such a price as abroad ... it's like buying a liter of Jameson in Duty Free for 700 rubles, while in Russia the cheapest I found for 1000 with a penny (wholesale supplier). A nice bonus for it was the built-in KVM module. Those. the server data at the server club were equipped with a KVM module, which meant that I could always and at any time connect to KVM, completely free. At one time I remember this appeared on SUN servers. It is worth noting that on March 31, in the caravan, when there was a full star, the queue at KVM reached several hours, while KVM remained a paid service. We could afford to come to the DC and figure it out on the spot with the server, which in fact simply destroyed part of the FS and expected fsck to start manually due to an emergency power outage. But what about, for example, regional customers? wait a few hours for KVM, pay for it and either run fsck manually or say goodbye to your FS? Caravan - you freaks.
In general, I was satisfied with the choice and instructed the administrator to transfer everything to Serverlab. The transfer was not without complications. Serverlab easily and quickly allocated servers to the fronts, but pulled with a powerful server under the base. At first he was not in the DC, then it turns out he was set up wrong. The guys immediately identified the “analogue” of a slightly less weak configuration, they say first here, but as soon as we set it up, you will move. Hemorrhoids, but in general it’s pleasant ... worrying ...
I was worried about ping for a long time, but in the end I saw that it was not so much different from Russia to stay in the country. For example, if in a caravan the average response time was 80ms, in the Server Club from American servers the response time was 200ms. A little more than 2 times, but for the user is completely invisible.
A few days later we completely moved and healed in Holland on high-quality gigabit American channels. The guys from Serverlab crossed themselves and continued their work. I am not the easiest client and have actively used an “individual approach” to clients. A low bow to the managers, they got to the fullest ... But we moved. Fortunately there was no limit. Suddenly ... competitors woke up. We are a rather unique service in RuNet. We have only 3-4 competitors that appeared later than us. We provide a taxi exchange service for taxi companies in Moscow, St. Petersburg and other regions and are the largest taxi exchange service. On the day that we did not work and the next day of unstable work, customers partially migrated to competitors. We are not offended, this is normal. Say, if mamba.ru closes, everyone will switch to loveplanet.ru and this is normal.
DDoS attacks went on us ...
Qrator.net
At first there was a child attack of up to 100 megabits. We looked, neighing. Then came 300 megabits of bots. The fronts survived without problems, but we felt some tensions associated with free disk space. Having made sure that we are dealing with DDoS attacks, we again breathed a sigh of relief. But it wasn’t here ... Theserverclub stat recorded 1 gigabit of incoming traffic. I couldn’t physically fix more statistics. The guys from the serverclub said that at that moment the total input to the server increased by 4-5 gigabits per second.
I have long thought to give protection from DDoS to professionals, but I wanted to defeat DDoS myself. By tuning the code and nginx, it generally worked out until the attack grew wider than the channel.
It was 10 p.m. And the qrator.net addresswas typed not the first time. The guys said that they work seven days a week, around the clock) it was a good way to check.
Earlier, I heard about the curator, more precisely about the Moscow State University, which for free protect the server from DDoS by studying attacks. Now they have grown into a paid curator.net service, which is generally logical. My first question, when I got through to them, was “Why are you bringing us?”. I really did not imagine how much an attack in 4 gigabytes and, most importantly, who has such resources can cost. Manager question plunged into a stupor. After half an hour, we had already agreed on everything and after about an hour stood up under their protection. The attacks did not stop for another two weeks, but they were no longer the main problem and went by the wayside.
The thing is that we turned out to be terrible clients not only for the server club, but also for the curator. Imagine ... One service, useful attendance of 500 (!!!) unique users per day. Not 500k, but 500. Less than classmates.km.ru 3 servers and a channel in gigabit. And just do not need to tell me about optimizing the code, the base, etc. I kept traffic at 1 million hits on nnm.ru on a relatively weak server at the time. And these 500 hosts had to be filtered from hundreds of thousands of bots.
The curator could not cope with his task, he kept blocking useful clients. Is it worth repeating that having only 500 useful hosts, everyone was important to us. The co-owners of the resource insisted on abandoning the “Curator”, but nevertheless the joint actions of our developers and researchers of the curator yielded results. At the moment (about a month), all our customers get service without any problems, while all bots are in the forest. The curator sometimes fixes attacks, but they do not affect the resource’s performance, which is insanely pleasant.
By the way, who did not use the services of the “Curator”, I will do a little “review”, at least of what interested me.
Unlimited to gigabits costs 17k rubles per month. In this case, under gigabit, I will introduce “useful” traffic. The site also indicates that an additional dedicated IP costs almost 2 times more expensive, it is not. In fact, an additional IP costs 5k rubles. Well, and accordingly, if you have several projects on one server, you do not need to start up for each IP. The curator’s IP is attached to your IP or IP array (2.5.10, not important). If you have 100 projects on one server, all 100 will be protected for 17k rubles per month.
However, a connection during a DDoS attack costs + 6k rubles. The guys explain this by the fact that they also pay for traffic, and until their classifiers learn to ignore left traffic, they pay for it. And, accordingly, if it exists initially, you cannot learn to filter it every second. There are no complaints.
Upset only paid APIs. The curator has an API, you can manage whitelists and blacklists. However, this pleasure costs money. 5k connection and 3k per month. When the curator blocked useful users, we were determined to take the API, but common sense suggested that paying for anti-DDoS + to prevent users from being blocked is not correct. As a result, he gave a command to developers at each POST authorization to add IP to the white list through a personal account. In your personal account, you can add IP to white lists.
Perhaps it is worth noting that the curator copes with standard projects with a bang. We supported and support the site of the film Burnt by the Sun 2: Citadel"And from the first days they hung them under the protection of the curator. We survived several serious attacks and not a single complaint from users about the inaccessibility of the project.
Today, May 16, a month and a half has passed since the moment we thought seriously and moved from the Russian DC to a foreign one. During this time we have experienced a lot. Anger and threats of “Caravan”, pleasant surprise at the prices of foreign DCs, DDoS attacks of 4 gigabits and experience with the “Curator”.
In fact, a pleasant experience in a short time. But seriously, it’s a shame when the famous Russian DC easily turns off the power without warning and considers this the norm. When the tsiska lease ends with the excuse “oh, we forgot to set it up”, while our service “lies”. When for a project with 500 hosts in Russia you need to think about a gigabit port and an AntiDDoS attack, because "Competitors are not asleep." And that you have to buy a voice recorder in case “we draw your attention to improve the quality of service, all conversations with the operator are recorded” is a lie.
