December 13, 2017 at 12:47Case Study on Backup Verification (SureBackup) and Using vSAN
- Transfer
Hi, Veeam Support is back with you! Today in the center of our attention is a very curious case: the SureBackup verification task stops in the middle of creating a snapshot, giving a general error message: “The specified feature is not supported by this version.” (This functionality is not supported in this version ).
The verification task included several virtual machines, but failed to process a single one. Moreover, the verification task was based on the backup task, which worked without any errors, performing a backup of this VM itself.
Read about our investigation under the cat.
In fact, such errors can occur for several reasons - for example, there are a number of limitations to the functionality of the free Free ESXi hypervisor (therefore, we do not support it). But this was not the case here - the hypervisor was clearly not Free, all other machines were perfectly processed by the SureBackup task, and the problem machine was normally backed up by the backup task.
Nevertheless, since this machine became a stumbling block for the SureBackup task, it made sense to consider it more closely.
The first thing that caught my eye was that this machine, let's call it VM01, was significantly larger than the others in size: one of its drives was larger than 2TB. Having noted this fact, we began to analyze the log of the job Veeam Backup & Replication. Here's what happened:
Honestly, at that moment log entries didn’t help us much, but at least it became clear that the VMware vSphere API was the generator of the error message.
Here we, without further ado, performed a search query, entering in it the line of the initial error message:
The disc name appeared in the found entry. It turned out to be the same VMDK disk larger than 2TB, which has already attracted attention. Also appears is the disk type - seSparse , i.e. space-efficient sparse .
Here we remembered that when creating a snapshot of a disk larger than 2TB for a delta disk, VMware vSphere does not use VMFS (vmfssparse), but just the seSparse format . This is written in the VMware KB 2058287 knowledge base article .
Our problem is probably somehow related to using a drive like seSparsefor storing the delta (redo log). But what exactly was the root cause? And here we drew attention to one more detail: the source VM and virtual laboratory involved in the SureBackup task were located on the vSAN datastore.
It is well known that SureBackup verification technology uses the Veeam vPower NFS datastore to “lift” virtual machines from it, while the redo logs for the machines in the verification task are redirected to the main datastore (in our case, it is vSAN):
That is, it was formed such a storage system in which the main disks were located on the NFS datastore, and the deltas (redo logs) on the vSAN. But since for all machines except the problem VM01, such a system worked fine, we came to the conclusion that we need to investigate the issue of compatibility of seSparseand vSAN.
The investigation led us to a VMware vSAN document that says: “Virtual SAN does not support SE Sparse disks.” - The virtual SAN does not support SE Sparse drives .
The document, however, was related to vSphere v5.5, and in our case it was vSphere 6.0. We contacted colleagues from VMware, who confirmed that this limitation applies to both version 6.0 and version 6.5.
Finally, all the pieces of the puzzle were put together, and here is the result: such a configuration of storage systems is impossible, where the main disk would be on NFS / VMFS, and the delS file (redo log) seSparse on vSAN - because, as the VMware documentation says , snapshot inherits the type VMFS_type.
An attentive reader will ask a reasonable question: why did backups work normally for large disks (more than 2TB)? After all, seSparse is not supported by vSAN? The answer is simple: when creating a VM snapshot, whose main disk is located on vSAN, VMware uses the VSANSPARSE snapshot type to save the delta .
Note: You can read more about VSANSPARSE in the VMware article ( downloadable PDF ).
Admin note:
We advised the user to transfer the "sandbox" (virtual lab) to the datastore without using vSAN. In this case, when redirecting, the type of snapshot will not change.
Similar problems can occur when working with VMs on vSAN if you try to run Instant Recovery Instant recovery for machines with large disks (> 2TB). Note that the above solution is applicable here, since a snapshot redirect is also performed.
Further studies of the issue showed that in some cases the problem also manifests itself when working in the hot-add mode (virtual appliance). In any case, you need to carefully analyze which datastores and what types of snapshots will be involved in a particular operation. Well, if you still have questions, we are always ready to help.
In anticipation of the winter holidays, Veeam is giving presents again: we are giving away 6 tickets to the VeeamON 2018 conference, which will be held in different regions of the world in the new year. The ticket includes a ticket to both sides, accommodation in a comfortable department near the conference venue, and, of course, attending all conference events (including those for which you need to book a place).
All you have to do to participate in the draw is register here .
If you read us, being, for example, in Australia or Canada, then you can open the page of the necessary regional conference, links to them here .
Registration will close on December 31, 2017, so it’s better to hurry up.
The names of the six lucky ones will be announced in the new year 2018. You can follow the news on our page on FB , as well as on Twitter .
The verification task included several virtual machines, but failed to process a single one. Moreover, the verification task was based on the backup task, which worked without any errors, performing a backup of this VM itself.
Read about our investigation under the cat.
In focus
In fact, such errors can occur for several reasons - for example, there are a number of limitations to the functionality of the free Free ESXi hypervisor (therefore, we do not support it). But this was not the case here - the hypervisor was clearly not Free, all other machines were perfectly processed by the SureBackup task, and the problem machine was normally backed up by the backup task.
Nevertheless, since this machine became a stumbling block for the SureBackup task, it made sense to consider it more closely.
The first thing that caught my eye was that this machine, let's call it VM01, was significantly larger than the others in size: one of its drives was larger than 2TB. Having noted this fact, we began to analyze the log of the job Veeam Backup & Replication. Here's what happened:
Honestly, at that moment log entries didn’t help us much, but at least it became clear that the VMware vSphere API was the generator of the error message.
Analyzing VMware Logs
Here we, without further ado, performed a search query, entering in it the line of the initial error message:
The disc name appeared in the found entry. It turned out to be the same VMDK disk larger than 2TB, which has already attracted attention. Also appears is the disk type - seSparse , i.e. space-efficient sparse .
Here we remembered that when creating a snapshot of a disk larger than 2TB for a delta disk, VMware vSphere does not use VMFS (vmfssparse), but just the seSparse format . This is written in the VMware KB 2058287 knowledge base article .
Our problem is probably somehow related to using a drive like seSparsefor storing the delta (redo log). But what exactly was the root cause? And here we drew attention to one more detail: the source VM and virtual laboratory involved in the SureBackup task were located on the vSAN datastore.
It is well known that SureBackup verification technology uses the Veeam vPower NFS datastore to “lift” virtual machines from it, while the redo logs for the machines in the verification task are redirected to the main datastore (in our case, it is vSAN):
That is, it was formed such a storage system in which the main disks were located on the NFS datastore, and the deltas (redo logs) on the vSAN. But since for all machines except the problem VM01, such a system worked fine, we came to the conclusion that we need to investigate the issue of compatibility of seSparseand vSAN.
The investigation led us to a VMware vSAN document that says: “Virtual SAN does not support SE Sparse disks.” - The virtual SAN does not support SE Sparse drives .
The document, however, was related to vSphere v5.5, and in our case it was vSphere 6.0. We contacted colleagues from VMware, who confirmed that this limitation applies to both version 6.0 and version 6.5.
Finally, all the pieces of the puzzle were put together, and here is the result: such a configuration of storage systems is impossible, where the main disk would be on NFS / VMFS, and the delS file (redo log) seSparse on vSAN - because, as the VMware documentation says , snapshot inherits the type VMFS_type.
An attentive reader will ask a reasonable question: why did backups work normally for large disks (more than 2TB)? After all, seSparse is not supported by vSAN? The answer is simple: when creating a VM snapshot, whose main disk is located on vSAN, VMware uses the VSANSPARSE snapshot type to save the delta .
Note: You can read more about VSANSPARSE in the VMware article ( downloadable PDF ).
Admin note:
- Snapshots of type VSANSPARSE are created on vSAN disks.
- Snapshots like VMFS_sparse or seSparse are created on regular VMFS disks, and the size of the disk and the version of VMFS are important (for example, snapshots on VMFS6 will always be of type seSparse, regardless of size).
- In the case of a redirect to another datastore, the snapshot disk type will be inherited from the parent disk.
We offer a solution
We advised the user to transfer the "sandbox" (virtual lab) to the datastore without using vSAN. In this case, when redirecting, the type of snapshot will not change.
Similar problems can occur when working with VMs on vSAN if you try to run Instant Recovery Instant recovery for machines with large disks (> 2TB). Note that the above solution is applicable here, since a snapshot redirect is also performed.
Further studies of the issue showed that in some cases the problem also manifests itself when working in the hot-add mode (virtual appliance). In any case, you need to carefully analyze which datastores and what types of snapshots will be involved in a particular operation. Well, if you still have questions, we are always ready to help.
In addition to the useful, here is the pleasant
In anticipation of the winter holidays, Veeam is giving presents again: we are giving away 6 tickets to the VeeamON 2018 conference, which will be held in different regions of the world in the new year. The ticket includes a ticket to both sides, accommodation in a comfortable department near the conference venue, and, of course, attending all conference events (including those for which you need to book a place).
All you have to do to participate in the draw is register here .
If you read us, being, for example, in Australia or Canada, then you can open the page of the necessary regional conference, links to them here .
Registration will close on December 31, 2017, so it’s better to hurry up.
The names of the six lucky ones will be announced in the new year 2018. You can follow the news on our page on FB , as well as on Twitter .