Automatically downloading the list of banned sites from the Roskomnadzor service

Platform: Windows XP / Vista / 7
Language: Perl The

service created by Roskomnadzor has a very long and difficult to read name: “A unified registry of domain names, web site pages on the Internet and network addresses that allow you to identify sites on the Internet” containing information the dissemination of which is prohibited in the Russian Federation ”, therefore we will call it“ Roskomnadzor ”service for simplicity.

Task
Organize automatic downloading of the lists of prohibited sites from the Roskomnadzor resource. The system must be implemented within one host, downloaded files must be rotated with a given depth.

Lyrics
Regarding the list of banned sites, it has already been said a lot and discussing this topic again does not make sense.

However, daily downloading of the list of banned sites, despite the obvious senselessness of this lesson, has become a necessity for many Internet providers. This is mainly done in order to avoid claims from Roskomnadzor. Naturally, the first thing that comes to mind is to automate the download process. Unfortunately, on the Roskomnadzor website, the automation method is described rather vaguely, and there is practically no technical support. In addition, in the case of automation of the process, work with the service is carried out according to the SOAP protocol, which also does not add joy.

For this reason, I want to introduce one of the possible ways of organizing the automatic download of the list of banned sites. It is possible that someone also had such a problem, and I hope my article will help those people who are currently looking for a solution to it.
I must say that a number of articles have already been written on this topic, including on Habré, the methods presented in these articles, for the most part, did not allow achieving full automation. Somewhere it was necessary with hands to form a request for a list, somewhere to run scripts with hands, etc. In my case, the system works completely automatically, I only sometimes see if everything is fine. Frankly, in order to raise the system, I used the article I foundon toaster. It presented scripts in Perl, which I changed a little and added something.

The implementation, of course, turned out to be rather crude, but quite working. All managed to be placed on one host. This is an old accounting computer under Windows XP, into which Rutoken was plugged in, connected to a network, and closed in a rack. Accordingly, the directory into which the archives are downloaded from the Roskomnadzor service is shared in the local network.

Now consider the implementation.

As the OS, as I said, Windows XP was used (which version does not matter), Windows Vista and Windows 7 are also suitable for these purposes.
Initially, we believe that the only purpose of the host is to work with the registry of prohibited sites.

1. Qualified detached electronic signature (ES) and software (software) for working with it


The acquisition of electronic signatures is a separate topic for discussion. You can buy it from various companies that have the appropriate license, how to do it, usually it is worth finding out from the company itself. In our case, the EP was acquired from the company LLC "CRIPTO-PRO", so this is the case I will describe.

The purchased kit included: a signature carrier (Rutoken) with a cryptographic key, an ES certificate, a certificate for technical support and a disk with a set of software.

When the OS is installed and the host is ready to work, the first thing to do is install the necessary software for working with ES.

First, install the drivers for Rutoken (those that were on the software disk did not fit). You can find them on the manufacturer's website (section Rutoken for Windows) .

Download drivers and install. If everything is done correctly, then after the Rutoken is inserted into the USB port, an LED will light on it. (The program is accordingly called the “Rutoken Control Panel”).

Next, install the Crypto-PRO program. You need to do this with a connected Rutoken.
The certificate is valid for one year, then it must be renewed. In this case, you must delete the old certificate and connect a new one. Information about certificates
can be found in “Start-> Crypto-Pro-> Certificates”.
If there are difficulties in this place, it is best to call technical support, they will explain everything quite correctly.
In my case, I was updating an outdated certificate. To do this, I went to the "Certificates" section , then
"Certificates-current user-> Personal-> Registry-> Certificates" , and deleted the old certificate manually, inserted Rutoken and rebooted the machine. The current certificate was picked up automatically.

To verify that the certificate is installed correctly, you can sign the file manually using the Crypto-AWP program, which is included in the kit.

To do this, install the program and create an arbitrary, preferably not empty file.
Turn on Crypto-AWP, then “Signature-> Sign”, select the file and go through the dialog, at the end select the certificate and click OK. If everything is done right. That in the directory specified in the dialog, the file filename.sig will appear . This file is a detached electronic signature.

The Crypto-AWP program does not participate in the system of automatic uploading of lists of prohibited sites.

The most important thing that we need is the utility csptest.exe , (ES verification can also be done with its help) which is part of Crypto-PRO.
This is a console utility for creating ES. It works only under Windows OS.

According to the developers, this utility is needed exclusively for tests and cannot be used in the combat system, because she has incomplete capabilities. For normal operation, another utility is used, which can work on a Linux server, but you have to pay for it separately.

Practice has shown that csptest.exe is quite suitable for solving the task.

2. Organization of automatic download


Work with the Roskomnadzor service takes place under the SOAP protocol .

Algorithm next

Every 5 minutes we check whether the download was performed for the current day (does the archive with the correct name exist)
 If yes, do nothing.
 If not, start the download.
   We create an actual request for
   downloading the list of banned sites. We create a signature file. We
   send a request to the server and get a request code (or an error message if the request is correct).
    If the request is correct, wait 5 minutes (the request is processed 1-2 minutes) and send the request code to the server.
    We accept the archive with a list of restricted files in binary mode
   If the request is incorrect, we get an error code and do nothing.

The unloading system consists of the following:
4 files, of which 3 are perl scripts and one .bat file type
Files: make_request.pl , get_register.pl , rotation.pl , roscomnadzor.bat Script

assignment.
make_request.pl - creates a request to the Roskomnadzor service
get_register.pl - sends requests to the Roskomnadzor service and processes the results of
rotation.pl
- rotates downloaded archives with a given depth roscomnadzor.bat - synchronizes the operation of scripts and utilities

System deployment


Create a working directory in the root of the C: // drive (or the other drive). I have it called roscomnadzor.
Inside it, we create 2 subdirectories: source and result.

Scripts must be placed in the source directory, the result of its operation will be located in the result directory, i.e. Archives with lists of prohibited sites.
After placing the scripts, you need to register the paths to the files from the root directory (or put them in config if you want to do everything beautifully).

Principle of operation


To complete the procedure, you must run the file roscomnadzor.bat

Listing roscomnadzor.bat

if not exist "D:\roscomnadzor\result\reestr_%Date:~6,4%_%Date:~3,2%_%Date:~0,2%.zip" (
rem Запускаем скрипт формирования запросов
perl D:\roscomnadzor\source\make_request.pl
rem запускаем утилиту для создания подписи
<путь к  csptest>csptest.exe -sfsign -sign -detached -add -in <путь к фалу>request.txt -out <путь к   файлу>request.txt.sig -my <имя владельца сертификата>@example.ru
rem запускаем скрипт для полученрия списка запрещенных сайтов
perl D:\roscomnadzor\source\get_register.pl
ren  D:\roscomnadzor\result\reestr.zip  "reestr_%Date:~6,4%_%Date:~3,2%_%Date:~0,2%.zip"
perl D:\roscomnadzor\source\rotation.pl
)

1. Creating a request file

The make_request.pl script is responsible for creating the request, the request file is created in the source directory

. The download request is an xml file of the following form:

yyyy-mm-ddT115:00:00.000+04:00Название организацииИНН организацииОГРН организацииcompany@cmail.ru


Listing make_request.pl

use POSIX qw(strftime);
my $date = strftime "%Y-%m-%d", localtime;
#Удаляем старые файлы с запросом и подписью если такие есть
unlink('<путь к фалу>request.txt');
unlink('<путь к фалу>request.txt.sig');
#Формируем xml запрос на сервер
$request=''.$date.'T115:00:00.000+04:00COMPANY NAME11111111111111111111111company@example.ru';
#Создаем в рабочей директории файл запроса
my $filename = '<путь к фалу>request.txt';
   open(my $fh, '>', $filename) or die "Не могу открыть '$filename' $!";
    print $fh $request;
   close $fh;


2. Create a signature file

To create the signature file, we use the same utility csptest.exe. It is called from the bat file.

D:\"Program files"\"Crypto Pro"\CSP\csptest.exe -sfsign -sign -detached -add -in <путь к фалу>request.txt -out <путь к фалу>request.txt.sig -my <имя владельца сертификата>@example.ru


where:
-my - Indicates the owner of the key;
-in - Specifies which file to sign. If the file is not in the csptest folder, then the full path must be specified .;
-out - Specifies the name of the signature file;

3. Dialogue with the Roskomnadzor service

The script get_register.pl is responsible for sending requests and processing the results .


Listing get_register.pl

   use MIME::Base64;
   use SOAP::Lite;
   use POSIX qw(strftime);
#-----------------------------------------------------------------------
#Создаем предохранители на случай, если файл запроса или файл подписи не существует
$key1=0;
$key2=0;
if (open(FP, '<путь к фалу>request.txt'))
 {$key1=1;}
if (open(FP, '<путь к фалу>request.txt.sig'))
 {$key2=1;}
#Если оба файла существуют-начинаем
if( ($key1==1)&($key2==1))
{
   #----------------------------
   #Читаем в буфер файл запроса
   open(FIL,"<путь к фалу>request.txt");
   while ($line = )
    {
     $ln.=$line;
    }
   close (FIL); 
   $req=$ln;
   #----------------------------
   #Читаем в буфер файл подписи
   open(FIL1,"<путь к фалу>request.txt.sig");
   while ($line1 = )
    {
     $ln1.=$line1;
    }
    close (FIL1); 
   $sig=$ln1;
   #----------------------------
   #Получаем описание методов джля работы с сервисом через wsdl-схему
   $soap = SOAP::Lite->service('http://vigruzki.rkn.gov.ru/services/OperatorRequest/?wsdl');
   $r = $soap->getLastDumpDate(); 
   #Отправляем запрос на сервер   
   @r = $soap->sendRequest($req, $sig); 
   $code = $r[2]; 
   #Ждем 5 минут и отправляем код запроса на сервер
   sleep 1, print "$_\n" for 1..300;   #300
   @r = $soap->getResult($code);
   #-------------------------
   #Принимаем и сохраняем в файл архив(принимаем в бинарном режиме)
   #my $date = strftime "%Y-%m-%d_at_%H-%M-%S", localtime;
   #my $date = strftime "%Y-%m-%d", localtime;
   my $filename = '<путь к фалу>reestr.zip';
   open(my $fh, '>', $filename) or die "Не могу открыть '$filename' $!";
    binmode $fh;
    print $fh decode_base64($r[1]);
   close $fh;
}


The result of the script execution (subject to server availability) is an archive with the name of the form: reestr_yyyy_mm_dd.zip

4.Rotation of results

Even with a low download frequency, sooner or later, all the space on the hard drive of the machine will be busy, which can lead to the most dire consequences. In order to maintain the system operability for a long time, it is necessary to organize rotation, i.e. as new archives are downloaded, old ones will be gradually deleted.

The rotation.pl script will turn off the rotation.

The depth of rotation determines how many archives will be in the result directory at the same time (given the logic, for what time interval to the past relative to the current day)

Listing rotation.pl

use POSIX qw(strftime);
use Time::Local;
#Глубина ротации
$birthtime =timelocal(localtime()); 
$interval = 0 + # 0 секунд
0 * 60 + # 0 минут
0 * 60 * 60 + # 0 часов
50 * 60 * 60 * 24; # и 20 дней
$then = $birthtime - $interval;
my $date = strftime "%Y_%m_%d", localtime($then);
#Убираем устаревший файл
unlink('<путь к фалу>reestr_'.$date.'.zip');


5. Scheduled startup

To run according to the schedule, I used the standard Windows
task scheduler schtasks ( http://www.windowsfaq.ru/content/view/83/57 ) The

time interval through which roscomnadzor.bat will be launched should be longer than the request timeout from the server

I I launched the system in mid-March, while everything works without failures.
Thanks for attention. I hope my article was useful to you.

Also popular now: