January 4, 2013 at 17:43Problems with the list of sites of Roskomnadzor. Telecom operators, be vigilant
Hi, Habr.
We already wrote how it turned out to be non-trivial to access the list of prohibited materials of Roskomnadzor to organize automatic monitoring of the blocking of the penetration of our networks and sites of our clients. Unfortunately, the story of the attack on the numerous rakes has a continuation, so we would like to share it. The described information will be primarily useful to telecom operators, who, as we monitor the situation, are resources on the Roskomnadzor list.
So, having started automatic monitoring in November, we began to wait for hour X - the appearance of a client resource in the list. On the one hand, of course, the pleasant thing is that you need to rush to contact the client and resolve the issue of deleting blocked information, it’s not enough, especially since we recall that they give 1 day to the reaction, which is written in the memo to the telecom operatorOn the other hand, all the tests that were carried out before were synthetic one way or another, and the developer inside was impatiently waiting for the automation to work on real data, joyfully notifying technical support and administrators by all possible means. After all, it is nice (for developers, not administrators) to see in the logs information that the server has crashed, but duplicating - it pulled the load on itself, that the master base fell off, but the replica replaced it, etc. In a word, it was just interesting to see the system in action.
And so, on December 27, we received an email from Roskomnadzor that the page of our client site www.tourister.ruhit the unfortunate list. The fact is that Turister.ru is such a social network for tourists, where people post reviews, reviews and life hacks about various trips. And in a blocked article it was told how to travel to Amsterdam and what grass is worth buying there. It is worth noting that the reason for the blocking is quite understandable and does not raise any special questions, so we quickly contacted the administration of the resource and as a result the article was deleted. Everything seemed to be happy and end, but we were alarmed by one point - our monitoring of prohibited resources over the entire time did not react to this situation. Those. in the staff list, reports about the next freshly loaded dump were sent to our helpdesk, but there was no information that tourister.ru was blocked:
Of course, this order of things greatly alarmed us and we began to check the downloaded dumps, because especially in case of a similar situation, we keep a history of loaded databases.
By the way, I would like to separately note one interesting feature about dumps. As you know, the site zapret-info.gov.ru provides an RPC interface for unloading the registry database. There are 3 methods declared in the documentation :
In fact, the following case is assumed:
We implemented it, but pretty quickly we noticed that each monitoring report contains a freshly downloaded dump, while the file size and the number of contained documents remain unchanged. A quick run of the files through the diff-tool showed that the files differ only in the timestamp value. In fact, the date of the last upload is the value of the current time, with minutes and seconds discarded. As a result, the whole idea of saving traffic and resources just doesn't work.
So, having looked at all the dumps over the past few days, we came to the conclusion that they simply do not have a record about page blocking on tourister.ru:
At the same time, the zapret-info web interface correctly displays information about blocking:
We, of course, wrote a letter with questions by zapret-info@rsoc.rubut haven’t received an answer yet.
As a conclusion, I would like to quote ourselves from a previous article on the topic:
So, dear telecom operators, be careful.
UPD: In the comments, the deputy head of Roskomnadzor Ksenzov Maxim Yurievich, Ksenzov , answers questions related to the work of zapret-info. Attention habra-users, let's not minus Ksenzov just because he is a representative of the organization, maintaining a registry that everyone hates here, because then we will lose the opportunity to hear answers and comments from the “other side”.
We already wrote how it turned out to be non-trivial to access the list of prohibited materials of Roskomnadzor to organize automatic monitoring of the blocking of the penetration of our networks and sites of our clients. Unfortunately, the story of the attack on the numerous rakes has a continuation, so we would like to share it. The described information will be primarily useful to telecom operators, who, as we monitor the situation, are resources on the Roskomnadzor list.
So, having started automatic monitoring in November, we began to wait for hour X - the appearance of a client resource in the list. On the one hand, of course, the pleasant thing is that you need to rush to contact the client and resolve the issue of deleting blocked information, it’s not enough, especially since we recall that they give 1 day to the reaction, which is written in the memo to the telecom operatorOn the other hand, all the tests that were carried out before were synthetic one way or another, and the developer inside was impatiently waiting for the automation to work on real data, joyfully notifying technical support and administrators by all possible means. After all, it is nice (for developers, not administrators) to see in the logs information that the server has crashed, but duplicating - it pulled the load on itself, that the master base fell off, but the replica replaced it, etc. In a word, it was just interesting to see the system in action.
And so, on December 27, we received an email from Roskomnadzor that the page of our client site www.tourister.ruhit the unfortunate list. The fact is that Turister.ru is such a social network for tourists, where people post reviews, reviews and life hacks about various trips. And in a blocked article it was told how to travel to Amsterdam and what grass is worth buying there. It is worth noting that the reason for the blocking is quite understandable and does not raise any special questions, so we quickly contacted the administration of the resource and as a result the article was deleted. Everything seemed to be happy and end, but we were alarmed by one point - our monitoring of prohibited resources over the entire time did not react to this situation. Those. in the staff list, reports about the next freshly loaded dump were sent to our helpdesk, but there was no information that tourister.ru was blocked:
Of course, this order of things greatly alarmed us and we began to check the downloaded dumps, because especially in case of a similar situation, we keep a history of loaded databases.
By the way, I would like to separately note one interesting feature about dumps. As you know, the site zapret-info.gov.ru provides an RPC interface for unloading the registry database. There are 3 methods declared in the documentation :
- getLastDumpDate - returns the timestamp of the last update of the unload from the registry.
- sendRequest - a method for sending a request, in response to which a result code is returned.
- getResult is the method of obtaining the result.
In fact, the following case is assumed:
- We make a request for a timestamp, and if it has changed the last time, we perform further actions. In principle, it’s logical - we save traffic and protect the server (s?) Of Roskomnadzor from unnecessary load. If the timestamp is unchanged, we monitor the resources for this dump.
- We form and send a request. We get the code.
- By code - we get a dump and already check our resources on it.
We implemented it, but pretty quickly we noticed that each monitoring report contains a freshly downloaded dump, while the file size and the number of contained documents remain unchanged. A quick run of the files through the diff-tool showed that the files differ only in the timestamp value. In fact, the date of the last upload is the value of the current time, with minutes and seconds discarded. As a result, the whole idea of saving traffic and resources just doesn't work.
So, having looked at all the dumps over the past few days, we came to the conclusion that they simply do not have a record about page blocking on tourister.ru:
At the same time, the zapret-info web interface correctly displays information about blocking:
We, of course, wrote a letter with questions by zapret-info@rsoc.rubut haven’t received an answer yet.
As a conclusion, I would like to quote ourselves from a previous article on the topic:
Sometimes the task associated with working with government agencies can intervene in a completely unexpected place.
So, dear telecom operators, be careful.
UPD: In the comments, the deputy head of Roskomnadzor Ksenzov Maxim Yurievich, Ksenzov , answers questions related to the work of zapret-info. Attention habra-users, let's not minus Ksenzov just because he is a representative of the organization, maintaining a registry that everyone hates here, because then we will lose the opportunity to hear answers and comments from the “other side”.