Python + vshadow + robocopy - synchronizing a directory with a remote machine

Not so long ago, I had the task of copying a database from a remote branch to receive in our office a full copy of it at the time of creation. In theory, daily copying. The first thing that came to mind is the rsync server to which you are copying, and already from it a copy is used later. A quick search on the bol brought a less adequate instruction right there .

The problem is that there is a Win system on the client machine. And the receiving side also uses a similar OS. Rsync server on Linux is an extra layer between two machines. Installing Rsync on a Win server is possible, but for other abstract reasons, is unnecessary. As a result, I came across my own directory synchronization script and, combining it with creating a shadow copy, created a new solution.

The resulting script is completely here. In the article, I will simply give the main milestones and the resulting functionality.

The script creates a shadow copy of the disk. Copies the specified directories to a remote resource via smb. The differences between the source and the resulting copy will be calculated. Generates a report on copying and received differences, and then sends a report by mail to the specified recipients.

As in the above article, to start, download the Volume Shadow Copy Service SDK 7.2 from the Microsoft website . We install on the machine and pull out the utilities we need.

We will need vshadow.exe and vshadow.pdb files from this SDK. Attention - they are different for 64-bit and 32-bit systems.

After installation, they can be found here:

64-bit -% PROGRAMFILES% \ Microsoft \ VSSSDK72 \ TestApps \ vshadow \ bin \ obj-chk \ amd64
32-bit -% PROGRAMFILES% \ Microsoft \ VSSSDK72 \ TestApps \ vshadow \ bin \ release -xp

We put it in a separate directory. I usually create the c: \ Windows \! Script directory for different tasks and it already has subdirectories. Everyone’s personal business here. The resulting two files and the script can be copied to the target machine and customized script for your tasks. But the Python necessary to execute the script will have to be installed on each client machine.

The script body itself starts from line 137. To set up copying goals and other related facilities, you need to define variables.

####################
##  Переменные        ##
####################
work_dir = 'C:\Windows\!Script\Backup'             # рабочий каталог скрипта. Изменить на тот в который выложенна утилита vshadow.exe и сам скрипт
source_disk = 'c:'
destin_disk = 'o:'
param_script = 'vs_generated.cmd'
# список каталогов для копирования
cwd_list = [['o:\\Windows\\!Script\\Backup', '\\\\master\\apps\\temp\\viv\\backup'],['o:\\ocs-ng', '\\\\master\\apps\\temp\\viv\\ocs-ng']]
# настройка почты
from_addr = 'admin@typa.ru'                                       # адрес отправителя отчета
tech_addr = ['user1@typa.ru', 'user1@typa.ru']           # список адресатов

In this case, this is the working directory in which the script is located, the target disk with the copy, the name of the disk to which the created shadow copy will be mounted, the name of the file into which the vshadow.exe utility saves the values ​​of its variables. Next, the cwd_list variable contains a list of directories that you want to copy and where to copy. The variable contains a list of two element lists. Note that to escape the slash in paths in Python, you need to put it down twice, and to specify a link to a network path, even four.

And completes the definition of email sending settings. A variable containing the address of the sender of the letter and a variable containing a list of recipients.

Actually, to define variables in this section is all that is required for successful work. Next, we hang up the execution of the script in the task scheduler, not forgetting to specify the execution with administrator rights. Without admin rights, the creation of a shadow copy does not occur. If we copy to a network resource in a domain, it is advisable to create a specially wired account in the domain for this. Accordingly, we set the rights to the target directory in the domain for this account to our preferences. Please note that the robocopy utility has many options for its launch. In this case, the source directory and the remote directory are synchronized completely, up to ntfs rights, audit, and the owner of the directory. In general, the network directory will have to tinker with setting the initial ntfs rights on the shared network resource.

Well, in conclusion, I will quickly go through the main milestones in the script.

From line 169, we create the shadow copy.

# проверяем наличие файла создания теневых копий
# без него работать не будем поэтому выход с ошибкой
if not(path.exists(getcwd() + '\\vshadow.exe')) and not(path.isfile(getcwd() + '\\vshadow.exe')) :
        exit(1)
proc = Popen('vshadow.exe -nw -p -script=' + param_script + ' ' + source_disk, shell=True, stdout=PIPE)
proc.wait()
out = proc.stdout.readlines()

Pay attention to the call parameter -script = '+ param_script, as a result of vshadow.exe, the values ​​of the variables are laid out in this file. We are interested in only SHADOW_ID_1, which contains the identifier of the created shadow copy. Actually, further we are engaged in reading the received file and pulling out the identifier value for further work. And after we found the identifier, we connect the shadow copy.

proc = Popen('vshadow.exe -el=' + SHADOW_ID + ',' + destin_disk, shell=True, stdout=PIPE)

Further, everything is routine. We iterate over the copy list. We set robocopy on it. We read the logs and generate a report. In conclusion, we send a report to all interested. Delete the shadow copy and the file containing the variables from vshadow.exe

PS By the way, if the definition file remains when you run this script, vshadow will not be able to overwrite it and the script will try to connect the old shadow copy by its id. In theory, in the future it will not hurt to check the presence of this file even at the beginning of work and delete it if necessary.

PPS Please note that directories are being copied. If you want to copy the entire disk, you need to slightly modify the sync procedure. When copying, a log is kept and the name of the log file is obtained from the name of the source directory. If you just specify a disk, a name, a log file, something like 'o: .txt' will be generated and the script will fail. In this case, it is better to add a check that all characters in the received file name are valid and delete the others.