August 19, 2015 at 05:39Backing up ESXi virtual machines with ghettoVCB scripts
In this article, I will step by step describe the configuration of automatic backup and an example of restoring virtual machines running on the ESX (i) platform using free ghettoVCB scripts . I will focus on version ESXi 5.x, but the same tools work on versions 3.5-6.x, though for earlier versions the settings are slightly different. Backup will be performed on the NFS server. A report on the work will be sent to the mail. During the backup, a snapshot of the virtual machine (including the working one) is taken, the VMDK disks of the machine are saved, and the picture is deleted.
The ghettoVCB project is well documented, but in the process of implementation there were nuances that resulted in this instruction. I hope this article will be useful to novice administrators.
First of all, prepare the NFS server to your taste, where backups will be made. In my case, these are FreeNAS (Freebsd 9.3) and ZFS dataset with deduplication and compression enabled, which significantly saves space. Setting up on the ESXi server is done via SSH from the root user . It is possible under another user with administrative rights, but then it will not work to run scripts for verification from the console. Let's get started.
1. To configure, you need access to the server via SSH, enable via vSphere client:
2. We take scripts from the github repository and put the contents on the server. Be sure to set the execution bit, otherwise the scripts will not work:
3. We will backup once a week, with a cycle of 4 weeks. Expired backups will be deleted. We write the appropriate config file:
If the ESXi server already has a connected nfs disk with the same coordinates (server / path), then the disk will not connect.
The body of the letter is formed while the script is running, and then sent by the utility
4. Create a list of machines that need to be backed up. You can get it using the command
5. Configure cron to perform a periodic task:
The system time goes to UTC, so you need to make an adjustment for the current time zone. In my case, +7 hours - the backup will start on Sunday at 1 am. Logs must be written (or redirected to / dev / null), otherwise the script may freeze when the buffer allocated for the user is full. The design
6. Restart cron :
7. To send mail, you need to add permission for outgoing traffic to the firewall of the ESXi server:
You can run a test backup of a machine with the name "vCenterUpdate" as follows:
Run manually for the list of machines:
Each copy of the machines will be stored in directories of the form:
And be a machine disk in * .vmdk format and a machine configuration file * .vmx .
ESXi settings made above will survive until the first reboot. To save the configuration, you need to do a number of actions.
1. Add commands to change cron settings to the boot script:
2. To save the firewall settings, we will create our own VIB package and install it on the server. To do this, use the VIB Author utility . Unfortunately, it is only for 32-bit architecture, so I had to use an lxc container. During installation, it can scary to swear on dependencies of the form:
But this does not matter, and the key
Prepare the directory tree for building the VIB package:
And create two files. The first is the description of our package:
And the second file is email.xml, the contents of which are given above. And it will be located along the path
We collect the VIB package:
If necessary, you can use my package (at your own peril and risk).
3. Copy our package to the ESXi server, install and check what happened:
And just in case, re-read the rules of the firewall.
This way you can collect and fix other useful server settings.
4. And finally, run the ESXi server settings manually in the backup:
Immediately you need to consider that a machine that was backed up on a hot one after recovery will be like after a crash - data loss inside the machine is possible.
1. We connect the backup storage to the target ESXi server via NFS, or simply copy the data over ssh there.
2. We create the configuration file "vms_to_restore" of this kind:
3. Run the test run:
And combat:
4. Rejoice.
The second option. Since the backup is a dump of the machine with the configuration file, you can simply:
1. Copy it somewhere to the ESXi server.
2. Correct the machine configuration file (* .vmx) by changing the name and location fields of the machine disk:
3. Through vSphereClient, go to the repository:
and add a new car to the list:
4. If the recovery server is different, then in the machine settings we change “Network Connection”.
5. At the first start, it will ask where the car was drawn from, you need to answer what you transferred.
If you answer that you cloned, it will change unique data, including the mac address of the network card.
That's all. GhettoVCB scripts also support other interesting options and copy options, so it’s useful to read the documentation. The method is far from ideal, but if you want cheap and cheerful, then it is quite workable.
Thanks for attention.
The ghettoVCB project is well documented, but in the process of implementation there were nuances that resulted in this instruction. I hope this article will be useful to novice administrators.
Backup setup
First of all, prepare the NFS server to your taste, where backups will be made. In my case, these are FreeNAS (Freebsd 9.3) and ZFS dataset with deduplication and compression enabled, which significantly saves space. Setting up on the ESXi server is done via SSH from the root user . It is possible under another user with administrative rights, but then it will not work to run scripts for verification from the console. Let's get started.
1. To configure, you need access to the server via SSH, enable via vSphere client:
Configuration -> security profile -> properties -> SSH2. We take scripts from the github repository and put the contents on the server. Be sure to set the execution bit, otherwise the scripts will not work:
# chmod u+x /ghettoVCB-master/ghettoVCB.sh
# chmod u+x /ghettoVCB-master/ghettoVCB-restore.sh
3. We will backup once a week, with a cycle of 4 weeks. Expired backups will be deleted. We write the appropriate config file:
# cat /ghettoVCB-master/4week.conf
VM_BACKUP_VOLUME=/vmfs/volumes/backup
DISK_BACKUP_FORMAT=thin
#количество хранимых бэкапов
VM_BACKUP_ROTATION_COUNT=4
POWER_VM_DOWN_BEFORE_BACKUP=0
ENABLE_HARD_POWER_OFF=0
ITER_TO_WAIT_SHUTDOWN=3
POWER_DOWN_TIMEOUT=5
ENABLE_COMPRESSION=0
ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP=0
ENABLE_NON_PERSISTENT_NFS=1
UNMOUNT_NFS=1
# NFS сервер, куда будут лететь бэкапы
NFS_SERVER=10.1.3.101
NFS_MOUNT=/mnt/backup/vmware
NFS_LOCAL_NAME=backup
# папка на NFS сервере (для каждого ESXi своя)
NFS_VM_BACKUP_DIR=autobackup/vm01
SNAPSHOT_TIMEOUT=15
#настройки логирования в почту
EMAIL_LOG=1
EMAIL_SERVER=mail.core.local
EMAIL_SERVER_PORT=25
EMAIL_DELAY_INTERVAL=1
EMAIL_TO=admins@mail.local
EMAIL_FROM=ghettoVCB@vm01.core.local
WORKDIR_DEBUG=0
VM_SHUTDOWN_ORDER=
VM_STARTUP_ORDER=
- VM_BACKUP_VOLUME - path on the ESXi server where the nfs partition will be mounted;
- DISK_BACKUP_FORMAT = thin - VMDK format of the disk that will be created during backup;
- VM_BACKUP_ROTATION_COUNT - the number of backups stored;
- POWER_VM_DOWN_BEFORE_BACKUP = 0 - do not turn off the machine before backup;
- ENABLE_COMPRESSION = 0 - do not press the data, leave it on zfs;
- ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0 - backup of machines with snapshots (recent versions of scripts can do this, but we don’t need it);
- ENABLE_NON_PERSISTENT_NFS = 1 - nfs drive will only be connected during backup;
- UNMOUNT_NFS = 1 - and then will be disconnected;
- NFS_SERVER and NFS_MOUNT - coordinates of the nfs disk;
- NFS_LOCAL_NAME - the name that will be assigned to the connected array (datastores identification);
- NFS_VM_BACKUP_DIR - the path where copies will be added (relative to VM_BACKUP_VOLUME);
- EMAIL_LOG = 1 - enable sending the report by mail;
- EMAIL_ * - mail parameter settings.
If the ESXi server already has a connected nfs disk with the same coordinates (server / path), then the disk will not connect.
The body of the letter is formed while the script is running, and then sent by the utility
nc. This can cause a failure on the part of the mail server, with the argument " Recipient address rejected: Improper use of SMTP command pipelining". It is necessary to exclude the corresponding check for ESXi servers (for postfix this will be reject_unauth_pipelining). 4. Create a list of machines that need to be backed up. You can get it using the command
esxcli vm process list. Each line of the list is one machine:# cat /ghettoVCB-master/week.list
win7
win10
vCenterUpdate
5. Configure cron to perform a periodic task:
# cat /var/spool/cron/crontabs/root
#min hour day mon dow command
#……………………….
3 18 * * 6 /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list > /var/log/ghettoVCB-backup-week-$((($(date +\%d)-1)/7+1)).log
The system time goes to UTC, so you need to make an adjustment for the current time zone. In my case, +7 hours - the backup will start on Sunday at 1 am. Logs must be written (or redirected to / dev / null), otherwise the script may freeze when the buffer allocated for the user is full. The design
$((($(date +\%d)-1)/7+1))gives the week number of the month, so we will not breed garbage. 6. Restart cron :
# kill $(cat /var/run/crond.pid)
# crond
7. To send mail, you need to add permission for outgoing traffic to the firewall of the ESXi server:
- create a configuration file with the contents:
# cat /etc/vmware/firewall/email.xmlemail outbound tcp dst 25 true false - re-read the firewall rules
# esxcli network firewall refresh # esxcli network firewall ruleset list | grep email email true # nc mail.core.local 25 220 mail.core.local SMTP Postfix quit 221 2.0.0 Bye
You can run a test backup of a machine with the name "vCenterUpdate" as follows:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -d dryrun -m vCenterUpdate
Conclusion:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -d dryrun -m vCenterUpdate
Logging output to "/tmp/ghettoVCB-2015-08-18_07-15-08-23516502.log" ...
2015-08-18 07:15:09 -- info: ============================== ghettoVCB LOG START ==============================
2015-08-18 07:15:09 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = /ghettoVCB-master/4week.conf
2015-08-18 07:15:09 -- info: CONFIG - VERSION = 2015_05_06_1
2015-08-18 07:15:09 -- info: CONFIG - GHETTOVCB_PID = 23516502
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/backup/autobackup/vm01
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_NON_PERSISTENT_NFS = 1
2015-08-18 07:15:09 -- info: CONFIG - UNMOUNT_NFS = 1
2015-08-18 07:15:09 -- info: CONFIG - NFS_SERVER = 10.1.3.101
2015-08-18 07:15:09 -- info: CONFIG - NFS_VERSION = nfs
2015-08-18 07:15:09 -- info: CONFIG - NFS_MOUNT = /mnt/backup/vmware
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 4
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2015-08-18_07-15-08
2015-08-18 07:15:09 -- info: CONFIG - DISK_BACKUP_FORMAT = thin
2015-08-18 07:15:09 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0
2015-08-18 07:15:09 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3
2015-08-18 07:15:09 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5
2015-08-18 07:15:09 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15
2015-08-18 07:15:09 -- info: CONFIG - LOG_LEVEL = dryrun
2015-08-18 07:15:09 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCB-2015-08-18_07-15-08-23516502.log
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_COMPRESSION = 0
2015-08-18 07:15:09 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0
2015-08-18 07:15:09 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0
2015-08-18 07:15:09 -- info: CONFIG - ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0
2015-08-18 07:15:09 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all
2015-08-18 07:15:09 -- info: CONFIG - VM_SHUTDOWN_ORDER =
2015-08-18 07:15:09 -- info: CONFIG - VM_STARTUP_ORDER =
2015-08-18 07:15:09 -- info: CONFIG - RSYNC_LINK = 0
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_LOG = 1
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_SERVER = mail.core.local
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_SERVER_PORT = 25
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_DELAY_INTERVAL = 2
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_FROM = ghettoVCB@vm02.core.local
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_TO = admins@mail.local
2015-08-18 07:15:09 -- info: CONFIG - WORKDIR_DEBUG = 0
2015-08-18 07:15:09 -- info:
2015-08-18 07:15:10 -- dryrun: ###############################################
2015-08-18 07:15:10 -- dryrun: Virtual Machine: vCenterUpdate
2015-08-18 07:15:10 -- dryrun: VM_ID: 588
2015-08-18 07:15:10 -- dryrun: VMX_PATH: /vmfs/volumes/ds3524_ds/vCenterUpdate/vCenterUpdate.vmx
2015-08-18 07:15:10 -- dryrun: VMX_DIR: /vmfs/volumes/ds3524_ds/vCenterUpdate
2015-08-18 07:15:10 -- dryrun: VMX_CONF: vCenterUpdate/vCenterUpdate.vmx
2015-08-18 07:15:10 -- dryrun: VMFS_VOLUME: ds3524_ds
2015-08-18 07:15:10 -- dryrun: VMDK(s):
2015-08-18 07:15:10 -- dryrun: vCenterUpdate.vmdk 40 GB
2015-08-18 07:15:10 -- dryrun: INDEPENDENT VMDK(s):
2015-08-18 07:15:10 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 40 GB
2015-08-18 07:15:10 -- dryrun: ###############################################
2015-08-18 07:15:10 -- info: ###### Final status: OK, only a dryrun. ######
2015-08-18 07:15:10 -- info: ============================== ghettoVCB LOG END ================================
Run manually for the list of machines:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list
Each copy of the machines will be stored in directories of the form:
autobackup/vm01/VMNAME/VMNAME-FULL_DATE/
And be a machine disk in * .vmdk format and a machine configuration file * .vmx .
Saving ESXi Server Configuration
ESXi settings made above will survive until the first reboot. To save the configuration, you need to do a number of actions.
1. Add commands to change cron settings to the boot script:
# cat /etc/rc.local.d/local.sh
#...................
/bin/kill $(cat /var/run/crond.pid)
/bin/echo "3 18 * * 6 /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list > /var/log/ghettoVCB-backup-week-$((($(date +\%d)-1)/7+1)).log" >> /var/spool/cron/crontabs/root
/bin/busybox crond
2. To save the firewall settings, we will create our own VIB package and install it on the server. To do this, use the VIB Author utility . Unfortunately, it is only for 32-bit architecture, so I had to use an lxc container. During installation, it can scary to swear on dependencies of the form:
# rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
error: Failed dependencies:
libc.so.6()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libc.so.6(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libc.so.6(GLIBC_2.3)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libc.so.6(GLIBC_2.4)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libdl.so.2()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libdl.so.2(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libpthread.so.0()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
libpthread.so.0(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
But this does not matter, and the key
rpm --nodepswill save us. Prepare the directory tree for building the VIB package:
# mkdir -p stage/payloads/payload1/etc/vmware/firewall/
And create two files. The first is the description of our package:
# cat stage/descriptor.xml
bootbank mailFirewall 5.0.0-0.0.1 Lelik.13a Custom VIB from Lelik.13a Adds custom firewall rule for mail sender to ESXi host false community true true false true false And the second file is email.xml, the contents of which are given above. And it will be located along the path
stage/payloads/payload1/etc/vmware/firewall/email.xmlwhere the path after " payload1" is the desired path on the target server. We collect the VIB package:
# vibauthor -C -t stage -v mailFirewall.vib
And check its contents
# vibauthor -i -v mailFirewall.vib
**** Info for VIB: mailFirewall.vib ****
VIB Format: 2.0.0
VIB ID: Lelik.13a_bootbank_mailFirewall_5.0.0-0.0.1
VIB Type: bootbank
Name: mailFirewall
Version: 5.0.0-0.0.1
Vendor: Lelik.13a
Summary: [Fling] Custom VIB from Lelik.13a
Description: Adds custom firewall rule for mail sender to ESXi host
Creation Date: 2015-08-12 09:47:07.199735+00:00
Provides:
mailFirewall = 5.0.0-0.0.1
Depends:
Conflicts:
Replaces:
mailFirewall << 5.0.0-0.0.1
Software Tags: []
MaintenanceMode: remove/update: False, installation: False
Signed: False
AcceptanceLevel: community
LiveInstallAllowed: True
LiveRemoveAllowed: True
CimomRestart: False
StatelessReady: True
Overlay: False
Payloads:
Name Type Boot Size Checksums
payload1 vgz 0 347 sha-256 69aa821faa4ccd5a5e34e487ecf6049aa6bf55652ffffbfaae1257e40610c405
sha-1 4d77e529c8da74e82d4aa4e816bcf193e29ab8de
If necessary, you can use my package (at your own peril and risk).
3. Copy our package to the ESXi server, install and check what happened:
# esxcli software vib install -v /tmp/mailFirewall.vib -f
# esxcli software vib list | grep mail
# esxcli network firewall refresh
-f" key is necessary . And just in case, re-read the rules of the firewall.
This way you can collect and fix other useful server settings.
4. And finally, run the ESXi server settings manually in the backup:
# /sbin/auto-backup.sh
Restore a machine from a backup
Immediately you need to consider that a machine that was backed up on a hot one after recovery will be like after a crash - data loss inside the machine is possible.
1. We connect the backup storage to the target ESXi server via NFS, or simply copy the data over ssh there.
2. We create the configuration file "vms_to_restore" of this kind:
# cat /ghettoVCB-master/vms_to_restore
#";;"
# DISK_FORMATS
# 1 = zeroedthick
# 2 = 2gbsparse
# 3 = thin
# 4 = eagerzeroedthick
# e.g.
"/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50/;/vmfs/volumes/local_vm01/;3;vCenterUpdate-restore"
- the path where the restored machine lies;
- the path where to restore the machine (a directory for it will be created);
- disk type of the recovery machine;
- new machine name (optional).
3. Run the test run:
# /ghettoVCB-master/ghettoVCB-restore.sh -c /ghettoVCB-master/vms_to_restore -d 1
And combat:
# /ghettoVCB-master/ghettoVCB-restore.sh -c /ghettoVCB-master/vms_to_restore -l /var/log/vms-restore.log
################## Restoring VM: vCenterUpdate-restore #####################
Start time: Fri Aug 14 06:05:06 UTC 2015
Restoring VM from: "/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50/"
Restoring VM to Datastore: "/vmfs/volumes/local_vm01/" using Disk Format: "thin"
Creating VM directory: "/vmfs/volumes/local_vm01//vCenterUpdate-restore" ...
Copying "vCenterUpdate.vmx" file ...
Restoring VM's VMDK(s) ...
Updating VMDK entry in "vCenterUpdate-restore.vmx" file ...
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50//vCenterUpdate.vmdk'...
Clone: 100% done.
Registering vCenterUpdate-restore ...
12
End time: Fri Aug 14 06:11:19 UTC 2015
4. Rejoice.
The second option. Since the backup is a dump of the machine with the configuration file, you can simply:
1. Copy it somewhere to the ESXi server.
2. Correct the machine configuration file (* .vmx) by changing the name and location fields of the machine disk:
displayName = vCenterUpdate-restore
extendedConfigFile = "vCenterUpdate-restore.vmxf"
scsi0:0.fileName = "vCenterUpdate-restore-0.vmdk"
sched.swap.derivedName = "vCenterUpdate-ff0c3749.vswp"
3. Through vSphereClient, go to the repository:
Configuration -> storage -> ПКМ -> "browse datastore"
and add a new car to the list:
ПКМ -> "Add to inventory" на файле *.vmx4. If the recovery server is different, then in the machine settings we change “Network Connection”.
5. At the first start, it will ask where the car was drawn from, you need to answer what you transferred.
If you answer that you cloned, it will change unique data, including the mac address of the network card.
That's all. GhettoVCB scripts also support other interesting options and copy options, so it’s useful to read the documentation. The method is far from ideal, but if you want cheap and cheerful, then it is quite workable.
Thanks for attention.