How a startup can fly out of the index overnight

    Hello, Habr!

    A year ago, we ran into a problem. If in a nutshell - we were hacked. Hacked creatively, so that we did not immediately understand what was going on. In the process of solving the problem, it turned out to master useful skills: compose abuse-calls and communicate effectively with hosters, including - foreign, more happy about the security of our resource, constantly monitor the availability of our site in the Yandex and Google index, and be careful about the little things. In the described case, the usual symbol “-” (hyphen, minus, dash, dash, wand, hypisominus) played a cruel joke with us.

    We faced the problem of hacking for the first time, maybe some things were obvious from the very beginning, but we did not see them, or our steps in finding a solution will cause a smile, or a misunderstanding among the pros. Or maybe our experience will be useful for those who are faced with problems in the field of information security.

    So, our main resource is located at: : it has its own permanent audience, an array of indexed pages, for example, a directory of organizations: and com / suppliers.php, where a total of more than 5 million pages are contained; different interactive, development perspective and more. To optimize the resource, development and experiments, we use a sandbox site with the same name, but on a different domain: . The sandbox is closed from visits from the outside, incl. and from search bots that index pages.

    The first alarm signals began to come in June 2013, when, upon requests, in the results of Google’s search results, our pages began to appear, but they belonged not to the main resource, but seemed to be like a sandbox. We checked robots.txt of the Komovsky site (sandbox), once again made sure that it was closed from indexing, and suggested that high-speed Google could index part of the pages at the moment of short-term lifting of restrictions.

    And a few days later there was a reason for more serious concern - the attendance of the main resource on the RU domain fell sharply, as a result - the number of incoming calls from site visitors fell. And again, a sandbox site surfaced in search engines, but with a circumstance from the realization of which cold sweat broke through us synchronously, we took a closer look and found that there was no dash in the name of the “sandbox”, i.e. instead of the address there was the address , it is surprising that we had not paid attention to this thing before. Services like whois expectedly showed that the site was registered just a couple of weeks ago, for a private person, the American domain registrar CLOUDFLARE and the Moldovan hoster Trabia-Network completed the picture so that evenit became extremely clear to the most inexperienced person in our company - this site is not at all a “sandbox” and has nothing to do with us .

    A detailed analysis showed that the clone website Moldavashka completely preserved our original design, structure and content of the pages of the main domain and subdomains. Contact details (address, phone numbers, details) also remained unchanged, except for the Skype login specified on the main page: instead of, appeared, although Skype itself categorically refused to know this login (i.e. this login did not has been registered).
    At the same time, our main domain began to slow down significantly, mainly because the clone continued to parse new pages from it and updated information in the main sections. Each “freeze” was accompanied by SMS from the useful utility Ping Admin (this utility ultimately played almost the most important role in solving the problem), constantly monitoring the site’s performance, the number of “disturbing” SMS messages from it about freezing was then measured in hundreds.

    As it turned out, the clone proxied all requests to us, and in the replies replaced all links to itself, adding advertising to the page code. The problem was solved by setting the .htaccess file, namely by adding the lines

    SetEnvIfNoCase REMOTE_ADDR "^ xxx \ .xxx \. [0-9] + \. [0-9] + $" REDIR = toredir
    RewriteCond% {REDIR} ^ toredir $ [NC]
    RewriteRule ^ (. *) $ $ 1 [R = 301, L]

    that sent all proxy requests from the address pool of the phishing site to a special page, the text of which stated that you are on a phishing site and can find us in a Google search by company name. I had to resort to such a trick due to the fact that the code proxying all requests to our site automatically replaced all links to a phishing domain.
    Also, a large number of requests from a phishing proxy highlighted the problem with optimizing SQL queries and some functions. Queries and functionality have been completely redesigned, while we used Xdebug and Webgrind as an analysis tool.

    Performance was relatively restored, but two problems remained:
    1. The clone site was still in the search engine index, not us.
    2. On the clone site, updates were still happening synchronously with the updates on our domain, but without obvious freezes, which added to us even more confusion.

    And there were more questions than answers. Here is one of them: if the search engines determined the full take of two sites - why do they drop out of the index a site that is clearly older (the domain is registered three years earlier than the clone), and not a freshly duplicate? There were many delusional answer options, for example, we suggested that the priority for search engines (in particular, Google) is always a site in the * .com domain zone, which is why our main RU-domain is “gone”. And also there was a correspondence with technical support of Google. Although the one-way flow of letters to Google support is difficult to call correspondence, we just tried to find out whether any sanctions were imposed on our site, which caused the departure from the index. There were no reply letters. And then it turned out that we were not looking for a solution there.

    The brightest thought in the ongoing chaos was the idea of ​​writing a letter to the provider where the clone site was hosted. Previously phoned with the Moldovan technical support (by the way, they speak pretty tolerant Russian), we drew up a complaint in Russian and English (such a requirement) and sent it by ordinary e-mail.

    _________________________________________ The

    letter began like this:

    Good afternoon.
    On June 13, 2013, the phishing site was registered . The
    data confirms that it is hosted by you:

    City: Chisinau
    Country: Moldova, Republic of
    IP Range: -
    Provider Name: ICS Trabia-Network SRL
    Whois Server Version 2.0
    Domain Name: ISTBUDGET.COM
    Whois Server:
    Referral URL:
    Status: clientTransferProhibited
    Updated Date: 13-jun-2013
    Creation Date: 13-jun-2013
    Expiration Date: 13-jun-2014


    The answer was not long in coming, for two hours without any bureaucratic delays, the clone site was blocked, when trying to go to the address of the clone site, Chrome reported that the specified site address does not exist. Unfortunately, the request to provide information about the owner of the clone site was refused and the offer to receive this data through an appeal to local security agencies. We celebrated the victory and went home, and in the morning ...

    And in the morning, to our displeasure, the clone site was back in service. Fortunately for us, this time it was hosted not in China on dubious proxies, but directly at its domain registrar in America. We already went according to the established scheme - we sent a complaint about abuse of a domain registrar and within a few hours the site was again blocked, this time - permanently.

    In general, the quick reaction of the provider and domain registrar was pleasantly pleased. Without additional questions and difficulties, we got the desired result. And then there was the expectation that 2-3 days would pass from the moment the clone “died” and our site would return to the search engine index again, however, this did not happen, there were still pages from the clone site in the SERPs, upon which Chrome regularly reported that the resource does not exist at this address. And again, we wrote letters to the support of Yandex and Google. It was not possible to create a letter in Yandex webmaster, therefore, they just wrote technical support in free form.

    The solution to the problem was found by chance and not at all where the efforts were made: the Ping Admin utility stopped sending SMS at times when the site was freezing. We paid attention to this, went into the personal account of the service, checked the balance (it was positive), saw that Ping Admin defines our site as inaccessible for several days now. We wrote a letter to the support service, reported an error, and in response received a letter where technical support reported that our main domain gives the 301th redirect for search robots to the address of the Komovsky parasite site, respectively, the robots received the main resource a signal that this domain is a non-main mirror of the new clone site.

    Thanks to the 301 redirect, the Yandex "mirror" »successfully glued our main domain with a new spurious domain, and that’s why the clone site began to be indexed and appeared in the search results instead of the main resource.

    We found and fixed the vulnerability, glad that we got off easily, removed the redirects that were registered by the wrong hands in our code. By the way, the parasitic code was found quite easily by calling eval (base64_encode ()).

    Within 30 minutes, our main domain returned to the Google index, a little later - to Yandex. And this is a completely different story.

    Also popular now: