Multilevel backup storage for the smallest

More recently, waves of WannaCry and its clones swept across the planet. And the problem of cryptographers has been facing system administrators for more than 10 years. Sooner or later - but all the implemented and implemented measures to protect against encryptors do not help, and yet there is a user who opens the letter, attachment and receives a full "bouquet". Also, a lot of "pleasant and exciting" hours get the system administrator.

And then everyone clearly begins to understand that backups are needed (many, different, in different places). Those. rule 3-2-1, coined and described by Peter Krogh, is highly desirable to comply with. This article is an example that helps to make the implementation of this rule real "on the knee" - without buying expensive equipment (in the context of austerity).

So - the conditions of the problem being solved:

• There is a small virtualization environment from Vmware (a couple of ESXi servers, vCenter, the cheapest license package - the initial Kit - in general this is not important for this article. Similarly, the article is suitable for Hyper-V);

• There are about a dozen virtual machines whose contents do not want to be lost if they are worked out by an automated Ransomware script;

• There is a backup system from Veeam (free edition, backups are done using PowerShell and Task Schedule).

Tasks:

• Make backup copies of servers once a day at night;

• Duplicate copies (copy to NAS server with FreeBSD + ZFS). By the way, snapshots are also made on ZFS, which are automatically deleted according to a given schedule (zfSnap + Cron);

• Have an offline copy of "backups" on removable media.

Implementation:

Since the main server, which makes backups from running virtual machines, is controlled by the Windows Server operating system (due to the fact that Veeam Backup works so far only on the basis of this OS), it was decided to use PowerShell to implement the tasks.

Solution of the backup synchronization task between the main server (Windows) and the NAS server (FreeBSD):

To solve the problem, a script was required that would run through the Task Scheduler and synchronize directory A with network resource B, accessible via the SMB protocol. At first I tried using robocopy - but the tests showed a very low speed of the resulting script. And I decided to implement this script on another tool.

A five-minute search and 10 minutes of tests showed a very viable and ready-made solution: powershell-synchronizing-a-folder

The script turned out to be gorgeous:

• Works with both local disks and network resources;
• Allows you to exclude certain files from the task;
• Allows you to synchronize files according to a given template
• It works at maximum speed (that is, how much iron and network can issue - with this speed synchronization also passes, unlike robocopy).

As a result, a bunch of tasks appeared in the Task Schedule of the form on the main server:

powershell.exe "C:\Scripts\syncfolder.ps1  -SourceFolder:G:\Backups\WEBAPPS -TargetFolder:\\192.168.0.232\backups$\WEBAPPS"

And the task of synchronizing backups after completing Veeam Backup tasks was solved (2 copies with a delta in time).

The solution to the problem of creating offline backups:

The idea itself is simple:

• We connect an external USB 3.0 hard drive of 2 TB to the server with Veeam Backup

• Most of the time we keep it Offline (and by doing this we protect ourselves from automated Ransomware);

• When the script completes, it transfers the disk to Online, makes a directory with the current date, copies the current backups to it, and at the end of execution transfers the disk to Offline again.

Implementation:

The starting point is the command: Get-Disk - we need to understand what drives we have in the system and whether we can see an external USB drive:

PS C:\Windows\system32> Get-Disk
Number Friendly Name                            OperationalStatus                    Total Size Partition Style
------ -------------                            -----------------                    ---------- ---------------
1      WDC WD30PURX-64P6ZY0                     Online                                  2.73 TB GPT
0      WDC WD10EZEX-60M2NA0                     Online                                931.51 GB GPT
2      WD Elements 25A3 USB Device              Offline                                 1.82 TB GPT

Now we need to put the link to the USB drive into a variable. To identify it, it is proposed to use the attribute "Friendly Name". If you prefer to use other attributes, print the full list (get-disk | select *). Or look at the list of available properties and methods (get-disk | get-member).

Total first part of the script:

# Find USB disk by FriendlyName
$mybackupdisk = get-disk | where {$_.FriendlyName -like 'WD Elements 25A3 USB Device'}

Next, you need to transfer the disk from Offline to Online, and also make sure that the disk is in Read-Write mode (sometimes, for some unknown reason, the disk became Read-Only after switching to Online. To find the disk number, use the Number property ($ mybackupdisk.Number ).

We get a piece:

# Make disk Online
Set-Disk -Number $mybackupdisk.Number -IsOffline $False
Start-Sleep -s 5
# Make disk Writeable (some times it ReadOnly after online - shit happens...)
Set-Disk –Number $mybackupdisk.Number -IsReadonly $False
Start-Sleep -s 5

To identify the drive letter, we will make the following feint - we will put a label (name) on the USB drive: VMUSBBACKUPS (either through Disk Manager, or using the Set-Volume command).

Next - using the Get-Volume command, we determine the letter of the connected USB-drive (after transferring it to Online):

# Find Disk Volume
$usbvolumename = Get-Volume | where {$_.FileSystemLabel -like 'VMUSBBACKUPS'}

And actually copying the necessary data to disk itself:

Create a directory with the current date in the name:

$date = Get-Date
$newbackupfolder = $date.ToString("yyyy-MM-dd")
 # Full Backup Fath
$createdirfullpath = $usbvolumename.DriveLetter + ":\" + $newbackupfolder
 # Create Backup Directory
New-Item -ItemType directory -Path $createdirfullpath -Force -Confirm:$false
Start-Sleep -s 2

Copy backups:

# Source Backup Dir (with backups)
$sourcebackup = "F:\Backups\VCENTER\"
# Copy to USB from Disk
Copy-Item $sourcebackup -Destination $createdirfullpath -Recurse
Start-Sleep -s 5

Another option - when we do not need to create new directories and copies each time - but rewrite the files with new versions - then we use the previously found script to synchronize directory A with B:

# Sync from HDD to USB:
C:\Scripts\syncfolder.ps1  -SourceFolder:F:\Backups\ -TargetFolder:$usbvolumename.DriveLetter:\VMs\
Start-Sleep -s 5

In any case, when you finish copying or synchronizing, it is highly advisable to reset the operations cache (from RAM to HDD / USB) with the command:

# Write USB Disk Cache before offline
Write-VolumeCache $usbvolumename.DriveLetter
Start-Sleep -s 5

And do not forget to transfer the disk from Online to Offline again:

# Place USB to Offline
Set-Disk -Number $mybackupdisk.Number -IsOffline $True

Results:

• Received backups in three places (Windows server, FreeBSD server, USB drive);
• Two types of storage (in balls and on a disk);
• One media of a different type is trialable. You can even have a couple of disks - and just 1 or 2 times a month to swap them (one in the safe). Since the USB disk in Offline mode is 95% of the time, it can always be pulled out of the server painlessly.

My stats:

• this scheme has been working for 6 months without failures;
• amount of synchronized data (compressed and deduplicated backups - from 500 to 700 GB);
• The synchronization time to a USB drive is 1 hour 20 minutes on average (1 time per week on weekends).

Full scripts can be downloaded from Google Disk: BackupExamples