Dealing with Veeam SureBackup

Backups need to be checked.

As an introduction, a simple story from a stormy youth. Everyone is familiar with a situation where there are no resources, but you need to store backups. At one time, two 500GB disks were used to store backup copies of their systems. As you might have guessed, when using RAID-1, the usable space was limited to the very 500GB, which was sorely lacking. It was decided to use Linux LVM, thereby gaining 1000GB of space, to the detriment of reliability.

Over the course of a year, or maybe more, backups were added to this resource, scripts reported that everything was fine, and the list of backups was pleasant to the eye when viewing the contents of the directory.

The time came X, and it was necessary to recover, surprise knew no bounds when one of the backups was damaged, because everything said that the backups are done, stored, and in general, everything is fine. As it turned out later, one of the disks in LVM began to crumble, part of the data could be restored, and the other part was gone. Never do as I did in my time.

Now there are already few companies that do not use virtualization, a little more companies that do not back up their virtual machines, and perhaps even more that do not check their backups.

For those who are interested in how Veeam checks its backups, I ask for a cat.

Suppose you already use the Veeam backup system in your virtual infrastructure and every day you receive a notification in the mail that all backups are complete, but, unfortunately, this does not mean that these backups can be used. And there are actually many reasons for this:

• As in the example from life, problems with disks can affect the backups located on them (the worst case, as for me);
• The operating system may not start if inconsistent backups are used;
• An application located on the system may not feel well after restoring the system.

Etc.

Veeam Backup & Replication has an interesting solution called SureBackup, which allows you to check backups, as well as some applications located inside systems. For those who are not familiar with this technology, you can read here . You can get this opportunity by having an Enterprise or Enterprise Plus license .

The SureBackup algorithm is pretty simple:

1. An isolated laboratory is created on any host in the virtualization cluster;
2. Using vPower NFS, the virtual machine from the backup is launched in this laboratory;
3. A series of automated tests are performed;
4. A test recovery report is sent.

VPower NFS technology allows you to run virtual machines on hypervisors directly from backup files.

List of tests that SureBackup can perform:

1. Checking the start of the virtual machine;
2. Checking Heartbeat operating system (Requires VMware tools in the guest operating system);
3. Check ping to virtual machine;
4. Running scripts to check applications inside the system (an account is required to access the guest OS).

And now about how to set it all up.

I use a test lab in which there is a VMware ESXi 5.5 cluster, Shared Storage, as well as a separate hypervisor on which the test recovery itself will be performed.
All addresses are fictitious, any matches are random.

SureBackup is configured directly from the Veeam BR console, and the first thing to do is prepare the Virtual Lab. You can find it on the Backup Infrastructure → SureBackup → Virtual Labs tab .

Now there are no laboratories, it is necessary to create a new one:


Having clicked on Add Virtual Lab, first of all we will be asked to somehow name our laboratory and indicate its description.


Next, you need to specify the host on which virtual machines will be launched and tested. I prefer to use a separate host from the productive cluster:


The next step is to choose one of the storages connected to the hypervisor, which will host the temporary files needed by Veeam for recovery, as well as the virtual machine necessary for testing and network settings in the virtual laboratory:


To interact with test virtual machines over a network, a Proxy Appliance installation is required. You can refuse to use this feature, however, in this case, many test features, such as network testing, access to virtual machines through the network will be unavailable. Proxy Appliance is a virtual machine that looks at one interface in a working network, and the rest, depending on the configuration, is isolated. This virtual machine deploys automatically when you create a virtual laboratory, you do not need to install the system and configure routing manually:


To correctly configure the Proxy Appliance, you must specify the network settings under which this virtual machine will be available for the Backup & Replication server.

You must specify:

The network from the virtual hypervisor switch that this appliance will use;
IP settings under which this appliance will be available for the BR server;



After specifying the Proxy Appliance settings as the next step, you need to configure isolated networks for our virtual lab. There are 3 options:

1. Basic single-host - automatic configuration of network settings. In this option, only one isolated network will be created, similar to the network that was specified in the appliance settings;
   
   
2. Advanced single-host - manual configuration of isolated networks. I think that most people use not one network for their virtual machines, but several networks divided into VLANs, respectively, for the correct test of a network of virtual machines with different network settings, we need to create several networks;
   
   
3. Advanced multi-host - Used if the virtual laboratory needs to be divided into several hosts, for example, for testing replicas. Uses the capabilities of VMware Distributed Switch (VDS).

Since I need to test virtual machines located on different networks, I will use the Advanced single-host option.

On the Isolated Networks tab, the system automatically creates one isolated network, similar to the network where the proxy appliance is located (Virtual machines running in the laboratory from backups will be connected to isolated networks and will not broadcast to production).

In my case, this is a V39 network and VLAN ID 39:


You need to add another network for virtual machines from another VLAN. When you click Add, a window will appear in which you need to select the Production Network that corresponds to any of the vSwitch networks and map it to Isolated Network in the virtual laboratory. In my case, I add the V30 network that my virtual machine uses from the backup:



As a result, the following rule is formed: Virtual machines whose network interface uses the V30 network with vSwitch will be connected to the test-vLab1 V30 network in the virtual laboratory, and virtual machines with the V39 network to the test-vLab1 V39 network, respectively:


The next step is perhaps the most interesting. Using a proxy appliance virtual machine, we can access virtual machines from outside, because proxy acts as a router between our network and the isolated one in which the test machines are located. Access is provided by masquerading the real addresses of the machines that are in the virtual laboratory, while the machines themselves work inside an isolated network with their real addresses.

By default, the proxy appliance creates only one network interface for the first isolated network. I will create two interfaces for my two networks V30 and V39:


First of all, I will change the settings for vNIC1 (the first proxy appliance virtual machine interface) to interact with the V39 isolated network. You can do this by clicking the Edit button. Initially, the settings are as follows:


And now the changed values ​​for my network:


The first field is the isolated network into which our proxy appliance interface will look.
The following is the address and mask by which the proxy will look into the given network, as Veeam writes, usually this address is similar to the gateway of this network. Virtual machines in an isolated network can communicate with each other through this gateway.

The next field is masquerading (spoofing). This works approximately as follows (if I understand everything correctly):

I use a class C network mask and the corresponding network for masquerading 192.168.100.XXX.

How does Veeam work in this case? When restoring a virtual machine to a test lab, it determines the address of the virtual machine. Suppose this address 10.10.10.113.
Then, since I am using the network mask / 24, the last octet is taken from this address, i.e. 113. A masked address 192.168.100 + 113 is generated. As a result, my machine is accessible from outside from this address.

To gain access to it, you need to update your routing table, where you must indicate that we will go to the address 192.168.100.113 through the gateway (which in our case is the Proxy Appliance) with the address 10.10.10.62.

As a result, for my two isolated networks, the following configuration is obtained:


We follow further to Ready to Apply.

Upon completion of the Virtual Lab, you can verify the total settings of the virtual laboratory, which will be obtained:


And, by clicking the Next button, the creation of our laboratory will begin. What is Veeam doing at this moment?

1. On a dedicated host, a resource pool is created, in which the Proxy Appliance and restored virtual machines will be located;


2. A virtual switch vSwitch is created with the name of our laboratory;

3. On this vSwitch virtual networks are created, which were indicated earlier. test-vLab1 V39 and test-vLab1 V30. As you can see, this switch does not use physical network adapters, which prevents attempts to exit virtual machines outside;


4. The Proxy Appliance virtual machine is deployed (the machine is located on the datastore, which was specified at the very beginning);

5. In our example, this virtual machine has 3 network interfaces. The first to connect to the production network. This interface is connected to vSwitch0. Two others for an isolated network from vSwitch1 (the more networks - the more interfaces):


At this stage, the creation of a virtual laboratory is completed.

The next step is to create the Appliaction Groups - a group of virtual machines that will be launched during testing. For example DNS, Domain Controller, Mail Server. In my example, everything is simpler, just two independent virtual machines without services.

In Veeam BR, go to the Application Groups tab and select Add App Group. We indicate the name of our group:


Next, you need to select the virtual machines for the test. The order in which machines are added matters, because it is in that order that SureBackup will launch them. It would be illogical to start the mail server first, and then DNS. To add a machine, click on Add VM. You can add a machine from a backup file, from a replica, or from Storage Snapshot:


In fairness, it is worth noting that the order of the machines can be changed using the Move Up and Move Down buttons.

If you select a machine and click the Edit button, you can specify a list of tests that will be performed on this machine. These are roles, scripts and startup options. In my case, I want to make sure that the machine will start and respond to ping, so I am only interested in the Startup Options tab:


Fields of interest:

1. Maximum allowed boot time - the time that SureBackup will wait for the system to start;
2. VM heartbeat is present - heartbeat from the virtual machine will be checked;
3. VM responds to ping on any network interface - the virtual machine responds to ping via network interfaces.

Change the settings as you wish. You can also configure test scripts and more, but I will leave this outside the scope of this article.

We save our group, after checking the settings.

Now we have a virtual laboratory and a group of virtual machines that need to be tested. To do this is quite simple - you need to create the SureBackup task on the Backup & Replication tab :

1. As elsewhere, we indicate the name of the task and its description;
2. On the next tab, select our virtual laboratory;
3. Next, select our Application Group;
4. Also, we can bind backup tasks directly to this task if Application Groups are not used;
5. On the Settings tab, you can specify the email address of the recipient who will receive a report on checking backups;
6. The Schedule tab indicates the schedule by which the scan task should be started. Scan tasks should not be launched simultaneously with backup tasks, otherwise the backup file will be blocked and one of the tasks will wait for the other to finish.

As a result, we have the SureBackup section in Jobs and our testing task:


It is time to start the task. And I’ll try to explain how everything works. When the task starts, we see a list of virtual machines for testing:


Testing of virtual machines will go in the order in which our Application Group is assembled.

1. Proxy Appliance is automatically launched;

2. The first virtual machine is published using vPower NFS. At this moment, an additional repository from our VM is connected to our hypervisor via the NFS protocol, and publication is underway from it:



3. This virtual machine is launched from the backup file:


4. Next, Veeam waits 600 seconds before the OS starts and tests start, as it is written in the logs:


From the log you can see that he found a network interface from the V30 network in the virtual machine and assigned the test-vLab1 V30 network to it. Next, he checks the heartbeat and tries to ping the determined address:


In the total log, you can observe that the tests on the first machine have completed, the second on the list of machines from the Application Group is initialized:

Note: the Veeam BR server automatically prescribes routes to masked addresses and, in principle, you can connect to it from them (Here is the same address 192.168.101.113 from the example about masquerading above):


At the end of the test, Veeam shuts down virtual machines in the reverse order and then publishes them from vSphere:



We receive the treasured email:


That's all for setting up and running SureBackup.

A little more about vPower NFS:

1. vPowerNFS is mainly used in tasks such as Instant VM Recovery and SureBackup;
   
   
2. vPowerNFS connects NFS storage to the hypervisor and does not disconnect it at the end of the work. This is done so that the next time you do not have to mount it again and waste time. If you disable the NFS datastore, the next time it will be mounted again;
   
   
3. vPowerNFS presents a virtual machine from a backup file. All changes that occur in the disk subsystem of the machine are transferred to the redo log, which is stored on the vPowerNFS server, or on a specially selected datastore, leaving the backup file unchanged. If redo log is stored on the vPowerNFS server, take care of sufficient disk space on this server;

In general, setting up SureBackup with the exception of network setup and masquerading should not cause problems. It turned out a little more than I expected, but this cannot be divided into parts.

Thanks for attention. Make and check backups.