October 10, 2016 at 07:55Free backup utilities with free ESXI
Somehow I got a few personal projects that required a relatively large amount of disk space - about 2TB. There were no suitable VPS (few people offer a lot of HDD space), so I took a dedicated server from OVH, installed ESXI 5.5 with a free license there and it worked.
After some time, with the development of projects, I began to configure admin features - monitoring, backup, and found out that there was a server in which I was promised Soft RAID, and on which the hoster (OVH) rolled up my ESXI image - without RAID! That is just 2 disks. Well, yes, now I know that ESXI does not support Soft RAID, only Hard. It became uncomfortable. Yes, and 2TB was not enough. In general, I took a bigger server for myself, with hardware RAID and put ESXI 6.0 there.
And two problems arose, the solution of which I will describe here:
I will say right away that both of these tasks are easily solved if there is at least a minimum paid ESXI license. The fact is that the “native” Backup API in the free version of ESXI is turned off. Therefore, you have to find other ways.
With a paid license, there is a migration option through vCenter . There is also a free version of Veeam Backup , which allows you to backup and transfer virtual machines from one system to another without the need to stop them. But with the free ESXI license, the current version - Veeam 9 - does not work at all .
There is also a solution from HP - VM Explorer , which has a free Free Edition.
VM Explorer 6.2 can work with free ESXI, but:
Another popular solution is the ghettoVCB open source project , but it seemed a little complicated for me to use, and the documentation looks a bit outdated. About this project already wrote here on Habré .
I am sure that there are many other options. It will be interesting to read the comments of experienced admins. Although I suspect that the experienced ones work where they bought the necessary licenses and do not worry ...
Here you can just mention:
If you have experience using these products - share in the comments.
I eventually decided to use 2 tools:
Prior to version 4.4, Xsibackup was on Github , but now (version 6.0.7) Xsibackup has been removed from Github, now you need to install it from the authors website.
In the free version:
» A full list of options (in English)
Locally, that is, on one host, everything works fine: backups are done using the ESXI native utility - vmkfstools . Everything is fast and thin disks remain thin. With hard drives, I got a speed of about 60MB / s.
However, when copying to a remote host, I found the same problem as with HP VM Explorer - the full size of the VM is copied, even if the drive is thin, and only the smaller part is used.
When I asked the authors what was the reason, they wrote that rsync is used for copying between hosts, and it is not smart enough to skip unallocated blocks of thin disks.
When testing, I found that with repeated backups, rsync practically does not reduce the copy time - the full VM size again goes over the network.
The authors wrote that they plan to file their own utility instead of rsync, which will be much faster. They plan to release before the end of the year.
In my case, the hoster guarantees a network speed of 250Mb / s (~ 31MB / s), but really between the two hosts in the same data center the backup worked for me at 10-20MB / s. I don’t know what’s the matter here - whether it slows down the network, rsync or something else - but the process takes too long.
Update : I found an article - according to their benchmarks, it turns out that the matter is slowed down SSH (on top of which rsync works), according to NFS it would be faster.
I am glad that as a result, the discs remain thin.
The process of transferring VMs between hosts looks like this for me:
In the same way, the VM backup from the host to your home also looks like so far. To do this, ESXI is spinning at home - so that ovftool can only transmit payload over the network.
The forums write that there seems to be a way to copy files to NFS with the sparse option so that only existing data is transferred, but I have not figured it out yet.
I did not find a way to do incremental backup.
While I do this all manually from the console, I transfer it to another host, make the first backup, but over time I think I’ll configure everything through the crowns. Maybe I’ll add here a couple of paragraphs on how to configure crowns. Original instructions here: 33hops.com/xsibackup-cron-how-to.html
Thus, now I have the first copy lying nearby, on the same server, and is available for fairly quick recovery.
The second copy is in my home, that is, as recommended , in a physically different place. To recover, you will have to upload over the network, which is much slower. But the likelihood of need for this is also quite low.
The full manual in English is here , and you can download it there. Ovftool can be installed on any computer, and control the hypervisor from it. And you can put it directly on the ESXI host, although this is not a supported feature.
Ovftool does not know how to copy a VM in hot mode, that is, it requires that the virtual machine be turned off. Therefore - the need for Xsibackup above.
Several features of Ovftool:
A few more features of Ovftool, only when running on ESXI:
Examples of using ovftool:
After some time, with the development of projects, I began to configure admin features - monitoring, backup, and found out that there was a server in which I was promised Soft RAID, and on which the hoster (OVH) rolled up my ESXI image - without RAID! That is just 2 disks. Well, yes, now I know that ESXI does not support Soft RAID, only Hard. It became uncomfortable. Yes, and 2TB was not enough. In general, I took a bigger server for myself, with hardware RAID and put ESXI 6.0 there.
And two problems arose, the solution of which I will describe here:
- Transfer virtual machines (some of which are about 1TB) from one server to another with minimal offline
- Make regular backups
I will say right away that both of these tasks are easily solved if there is at least a minimum paid ESXI license. The fact is that the “native” Backup API in the free version of ESXI is turned off. Therefore, you have to find other ways.
With a paid license, there is a migration option through vCenter . There is also a free version of Veeam Backup , which allows you to backup and transfer virtual machines from one system to another without the need to stop them. But with the free ESXI license, the current version - Veeam 9 - does not work at all .
There is also a solution from HP - VM Explorer , which has a free Free Edition.
VM Explorer 6.2 can work with free ESXI, but:
- When creating a backup, the full image size is copied from the server, even if the disk is thin there. That is, if the disk of the virtual machine is 500GB, and only 50GB is recorded there, then 500GB will be copied. And so - every time. The incremental backup mode (only on the local computer) is in the paid version, I have not tested it - I know how effective it is.
- A free license allows you to backup only to local drives. That is, to copy to another ESXI host, you already need a paid license.
- There is no scheduler in the free version, that is, you need to start backups manually.
Another popular solution is the ghettoVCB open source project , but it seemed a little complicated for me to use, and the documentation looks a bit outdated. About this project already wrote here on Habré .
I am sure that there are many other options. It will be interesting to read the comments of experienced admins. Although I suspect that the experienced ones work where they bought the necessary licenses and do not worry ...
Here you can just mention:
- Nakivo , which has a 2 VM limitation in the free version.
- Unitrends , which has a 1TB limit in the free version.
- Thinware
If you have experience using these products - share in the comments.
I eventually decided to use 2 tools:
- Xsibackup, which has a free version
- Ovftool
Xsibackup
Prior to version 4.4, Xsibackup was on Github , but now (version 6.0.7) Xsibackup has been removed from Github, now you need to install it from the authors website.
In the free version:
- Hot backups, without stopping virtual machines. This is done using snapshot (snapshot)
- Configuring Cron in ESXI
- Email Reports
- Backup rotation
- Backup to another ESXI host. In the free version - using rsync, sharpened under ESXI. In the paid version there are also incremental backups ( OneDiff ) through the creation of intermediate snapshots (as for me, this is not a very good solution) and deduplication using their NAS ( XSINAS )
Xsibackup Installation Instructions
The same instruction in English - 33hops.com/blog_xsibackup-quickstart.html Xsibackup is
installed on the ESXI host from which you need to backup.
On ESXI, SSH must be enabled.
Register on the authors website - Download xsibackup - 33hops.com/xsibackup-vmware-esxi-backup.html
You will receive a free key and a script for installation on ESXI by email:
secretKey you will have your own.
If your datastore is called differently, then you need to register your path.
Seeing wget, someone can shake his head, and say that putting someone else’s software on the ESXI host is not secure, etc. However, for any backup, you will give root the password to the backup program, that is, you will trust someone in any case. When copying locally, Xsibackup uses only shell scripts that can be viewed and checked ...
Then create a folder where we will store backups - locally, or on another server:
If you copy backups between hosts, then we share SSH keys:
If we want backups to be launched through crowns, then:
We test that everything works locally:
To test the operation between hosts, we change:
If SMTP requires TLS, then --smtp-sec = TLS is supported.
installed on the ESXI host from which you need to backup.
On ESXI, SSH must be enabled.
Register on the authors website - Download xsibackup - 33hops.com/xsibackup-vmware-esxi-backup.html
You will receive a free key and a script for installation on ESXI by email:
cd /vmfs/volumes/datastore1/xsi-dir 2>/dev/null || mkdir /vmfs/volumes/datastore1/xsi-dir && \
cd /vmfs/volumes/datastore1/xsi-dir && \
esxcli network firewall unload && \
wget http://a.33hops.com/downloads/?key=64cG...secretKey -O xsibackup.zip && \
unzip -o xsibackup.zip || cat xsibackup.zip && echo "" && \
chmod 0700 xsibackup* && \
rm -rf xsibackup.zip && \
esxcli network firewall load
secretKey you will have your own.
If your datastore is called differently, then you need to register your path.
Seeing wget, someone can shake his head, and say that putting someone else’s software on the ESXI host is not secure, etc. However, for any backup, you will give root the password to the backup program, that is, you will trust someone in any case. When copying locally, Xsibackup uses only shell scripts that can be viewed and checked ...
Then create a folder where we will store backups - locally, or on another server:
mkdir /vmfs/volume1/datastore1/backupIf you copy backups between hosts, then we share SSH keys:
./xsibackup --link-srv=[second.esxi.system.ip]If we want backups to be launched through crowns, then:
./xsibackup --install-cronWe test that everything works locally:
./xsibackup --backup-point=/vmfs/volumes/datastore1/backup --backup-type=running --mail-from=email.sender@yourdomain.com --mail-to=email.recipient@anotherdomain.com --smtp-srv=smtp.yourserver.com --smtp-port=25 --smtp-usr=username --smtp-pwd=password --test-mode=trueTo test the operation between hosts, we change:
--backup-point="IP-OF-ESXI:22:/vmfs/volumes/datastore1", где 22 - это SSH порт.If SMTP requires TLS, then --smtp-sec = TLS is supported.
» A full list of options (in English)
Locally, that is, on one host, everything works fine: backups are done using the ESXI native utility - vmkfstools . Everything is fast and thin disks remain thin. With hard drives, I got a speed of about 60MB / s.
However, when copying to a remote host, I found the same problem as with HP VM Explorer - the full size of the VM is copied, even if the drive is thin, and only the smaller part is used.
When I asked the authors what was the reason, they wrote that rsync is used for copying between hosts, and it is not smart enough to skip unallocated blocks of thin disks.
When testing, I found that with repeated backups, rsync practically does not reduce the copy time - the full VM size again goes over the network.
The authors wrote that they plan to file their own utility instead of rsync, which will be much faster. They plan to release before the end of the year.
In my case, the hoster guarantees a network speed of 250Mb / s (~ 31MB / s), but really between the two hosts in the same data center the backup worked for me at 10-20MB / s. I don’t know what’s the matter here - whether it slows down the network, rsync or something else - but the process takes too long.
Update : I found an article - according to their benchmarks, it turns out that the matter is slowed down SSH (on top of which rsync works), according to NFS it would be faster.
I am glad that as a result, the discs remain thin.
The process of transferring and backup VMs
The process of transferring VMs between hosts looks like this for me:
- First, I make local backups using Xsibackup.
- I am registering a new VM - a copy from the previous step.
- Optional: You can "clean" the VM from deleted files with the command:
vmkfstools --punchzero VMname.vmdk
specify the main file, not the one that is -flat. - Then using Ovftool (see below) I copy to another host. Ovftool can send thin disks so that only the used part is sent.
- After that, the VM on the first host shuts down, and the new one starts up.
In the same way, the VM backup from the host to your home also looks like so far. To do this, ESXI is spinning at home - so that ovftool can only transmit payload over the network.
The forums write that there seems to be a way to copy files to NFS with the sparse option so that only existing data is transferred, but I have not figured it out yet.
I did not find a way to do incremental backup.
While I do this all manually from the console, I transfer it to another host, make the first backup, but over time I think I’ll configure everything through the crowns. Maybe I’ll add here a couple of paragraphs on how to configure crowns. Original instructions here: 33hops.com/xsibackup-cron-how-to.html
Thus, now I have the first copy lying nearby, on the same server, and is available for fairly quick recovery.
The second copy is in my home, that is, as recommended , in a physically different place. To recover, you will have to upload over the network, which is much slower. But the likelihood of need for this is also quite low.
Ovftool
The full manual in English is here , and you can download it there. Ovftool can be installed on any computer, and control the hypervisor from it. And you can put it directly on the ESXI host, although this is not a supported feature.
Install Ovftool on ESXI
In general, the process is this: first Ovftool is installed on Linux x64 (I did on Ubuntu 16), and then the files are transferred to the ESXI host.
Here are the steps:
Here are the steps:
- Register with VMware and download “VMware OVF Tool for Linux 64-bit”
- Run the downloaded file on Linux and then copy the resulting files to ESXI:
sudo /bin/sh VMware-ovftool-4.1.0-2459827-lin.x86_64.bundle scp -r /usr/lib/vmware-ovftool/ root@esx.com:/vmfs/volumes/datastore1 - Then, on the host itself, you need to edit one file - / vmfs / volumes / datastore1 / vmware-ovftool / ovftool and replace #! / Bin / bash with #! / Bin / sh in it
Ovftool does not know how to copy a VM in hot mode, that is, it requires that the virtual machine be turned off. Therefore - the need for Xsibackup above.
Several features of Ovftool:
- I used to have errors popping up: “The task was canceled by a user” or “Error: vim.fault.FileNotFound”. The cause of both errors was that the CDROM on the original machine was pointing to an ISO that was not on the target host. Try to guess about it from the error text ... It was decided by removing the CDROM device (it was used only for installing the OS).
- I have not tested it myself, but ovftool seems to not save snapshots.
A few more features of Ovftool, only when running on ESXI:
- Although ovftool runs locally, the path to the host must be specified completely, along with the user and password.
- It cannot read the password interactively from the console, but requires that it be passed in as a parameter on the command line.
- Only letters, numbers and `-` can be used in the password. When trying to use other characters, such as `#` - the password was not accepted.
Examples of using ovftool:
- List all registered virtual machines in the datastore:
ovftool -ds=datastore1 "vi://root:password@esx1.com/" - Copying a virtual machine (it must be registered):
ovftool -ds=datastore1 -dm=thin "vi://root:password@esx1.com/vmName" "vi://root:password@esx2.com/"