May 15, 2012 at 10:38Effective backups in Amazon Web Services - recipes
Hello!
Today we’ll talk about techniques for configuring file backups and MySQL / InnoDB / XtraDB in applications deployed in the cloud, using Amazon Web Services as an example.
During the development of the Bitrix24 cloud service, we tried several backup schemes, stumbled upon some pitfalls of Amazon architecture and software limitations - however, everything was
sorted out and buzzed :-) We also carefully consider the issue of incremental backup of large enough data volumes (hundreds of gigabytes and more ), raids and configurations with InnoDB / XtraDB.
But first of all, we’ll take a look at the data storage technologies offered by Amazon.
So, let's start with what Amazon offers us for storing virtual machine data - virtual block devices EBS . Such a disk is easily created and connected to the server in 2 clicks. The maximum disk size is 1TB. By default, there is a limit on 5000 drives and 20TB, but it is increased at the first request .
The technology of local block devices is also proposed, the data on which ... disappear with the server (and this can easily happen when the machine crashes) - but I won’t write about it, because we did not experiment with her.
You can see right away that they are slower than the iron ones. Saturation of the device (% util, iostat command) with a random read volume of a dozen or two MB / s (even less by write) quickly approaches 100%. Slowing down is clearly seen on popular operations such as copying folders from disk to disk, unpacking archives, etc.
In order to adequately start working with Amazon discs, the easiest way to “shove” them into a software raid. For databases, we use raid-10 on 8 EBS disks on both ext4 and xfs. A software raid is done quite simply , it works for a long time and practically does not break.
A raid can be especially useful if the EBS drive suddenly crashes.
Nevertheless, for some tasks we do not use raids - for example, for storing the MySQL binary log, for backups, etc. And for storing the nginx cache, raid0 on EBS disks worked well, which has been working steadily without failures for about a year.
To be honest, for a year and a half of working with Amazon EBS-disks, they never let us down (no stupid things like “bads”, reading errors, etc.) ... except for the case when lightning hit the Irish data center, then several disks flew at once from raid-10 :-)
However, if you carefully read what Amazon writes about the reliability of their disks, you understand that you also need to do the raid and, of course, regular backups:
Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% - 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.
On the other hand, we have more than a hundred loaded EBS disks in production and in a year and a half, software raids have never knocked out disks due to IO errors. I’m sure we would have changed more than one device on the “iron” drives, so draw conclusions.
When the data is relatively small and they do not change often, you can play around with tar. But imagine a large online store that stores business information both in the database and in files on disk: new files appear every minute, and the total content size is hundreds of gigabytes.
DRBD? Yes, but they did not try this technology in Amazon, and I often hear from colleagues about its incredible braking when errors occur.
LVM and snepshoty mode copy-on-write - the similarity of this technology, only with extra buns and offers us the Amazon. Snapshots of a block device can be done as many times as necessary. Wherein:
All this means that we can take snapshots of a huge folder of frequently changing content on the disk at least once every 5 minutes - they will be stored securely in S3 and if you need to roll back 1TB of mutable data 5 minutes ago - we can do this with ease:
Of course, it is technically impossible to instantly transfer 1TB of data from S3 to the SAN where EBS disks live, therefore, although the block device becomes available to the operating system, data will be poured onto it in the background for a certain time - therefore, the speed of working with the disk at first will not be very high. But, nevertheless, you must agree how convenient it is to make an incremental backup of a large amount of data and roll it back to any point, for example, a week ago with a step of 5 minutes? :-)
In addition to the ability to create snapshots from EBS-drives, you can send files to S3 directly. Easy to use utility s3cmd- you can synchronize the filesystem trees with the cloud in both directions (only changes are transferred based on the calculation of md5 on the local disk and storage of the md5 object inside s3 in “ETag”). We tried solutions based on FUSE - s3fs technology , but noticed slowdowns and long-term freezes with the growth of LA during its intensive use.
As I wrote above, EBS disks show adequate performance if they are combined into raid0, raid10. How to backup a raid? Snapshot each drive in turn? :-) We understand that it is impossible and the Amazon here does not offer us anything.
Good people have written a handy utility - ec2-consistent-snapshot . You can use it, or you can repeat its logic in scripts.
If you have xfs, you can use the xfs_freeze command .
To connect a saved raid, it is better to write a script that connects the disks to the machine and launches a software raid from them. The raid saved in snapshots by the above method rises beautifully without losing the file system log - we use it in different places in production.
So, we learned how to snapshot raids in s3 with any amount of data with a frequency of at least once every 5 minutes and restore data from them. I’m sure that many of these things will come in handy on various projects in the cloud.
Sometimes it’s more convenient not to bathe separately with each raid, but to make a snapshot of all the disks of the machine with one command . You can make a snapshot in 2 modes: with a car stop and without a stop. In the latter case, we are logically warned about a possible “corruption” of data on disks / raids:
When taking a snapshot of a file system, we recommend unmounting it first. This ensures the file system metadata is in a consistent state, that the 'mounted indicator' is cleared, and that all applications using that file system are stopped and in a consistent state. Some file systems, such as xfs, can freeze and unfreeze activity so a snapshot can be made without unmounting.
After creating a machine snapshot, an AMI (Amazon Machine Image) object appears, with links to the saved snapshots of each drive. You can run a server with all disks / raids from this object with one command - AWS API call RunInstances . Did you feel the power of technology? Working servers can not only be backed up entirely, but also lifted from the backup TOTAL with all raids with one team! This technology saved us dozens of hours of system administration during the Amazon accident in August last year - we took the machines out of snapshots and deployed the configuration in another data center.
However, there is a serious pitfall - the CreateImage command is completely opaque and it is unclear how long it takes snapshots from all server disks - a second or 10 seconds? The method of scientific poking was chosen interval - 5 seconds, allowing you to shoot complete images of the machine with raids. I warn you - thoroughly test the script before launching into production - however, before the “goodies” of the technology for creating a full backup of the machine, you must admit that it’s hard to resist :-)
Let me remind you of our task - to backup a project with traffic of millions of hits per day and hundreds of gigabytes of content that changes quite often (the heaviest content has been moved to s3 and downloaded separately). I will repeat the known reasonable approaches to MySQL backup:
To be able to quickly roll back a large online store 5-10 minutes ago in the event of a catastrophic deletion of data in the database (an erroneous request that kills data in several order tables - who else was not with? :-)) - it seems that only 3 option will work . However, as it turned out, the binary incremental backup upon creation exerts a considerable load on the already weak EBS disks, but it can take several hours to restore increments to the basic binary backup during recovery ...!
I do not consider here recovery scenarios from a logical backup with preliminary editing of the MySQL binary log - it is still not fast to do this.
And here the Amazon helps us again. MySQL incremental backup is done like this:
Now we have an AMI object with a hot MySQL backup and we have made the maximum possible to start it from the backup as quickly as possible.
Thus, it turned out to be pleasant just to make an incremental backup of the MySQL server in S3 with a frequency of at least once every 5 minutes and the possibility of its quick entry into production. If the server is used in replication, then it will restore it as a rule without any problems, if you did not forget to set conservative replication settings in the settings (well, or you can return it to work pretty quickly manually):
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1
sync_master_info = 1
sync_relay_log = 1
sync_relay_log_info = 1
For the system administrator, there are convenient utilities that pull the Amazon REST methods. Several utilities are downloaded for each web service used, and calls to utilities are scripted in bash. Here is an example of a script that changes hardware on a server:
For developers, there are libraries in different languages for working with the Amazon API. Here is the library for working from PHP - AWS SDK for PHP .
As you can see, scripting work with Amazon objects is simple.
Using concrete examples diluted with the theoretical base of popular practices and salted with effective Amazon API tools, we learned:
In one of the following articles I will tell you how to effectively manage traffic and balance the load between clusters and data centers of Amazon.
In practice, we actively use the technologies listed in the article, we experiment, are open and ready to share our experience and look for simple and effective solutions to emerging problems using the tools of the open source world.
I invite everyone to register on our Bitrix24 cloud service and see what happened. I also invite everyone to the FREE seminar on web clusters and high loads, which will be held in the conference room of 1C on May 22.
Good luck to everyone in building scalable and fault-tolerant systems!
Today we’ll talk about techniques for configuring file backups and MySQL / InnoDB / XtraDB in applications deployed in the cloud, using Amazon Web Services as an example.
During the development of the Bitrix24 cloud service, we tried several backup schemes, stumbled upon some pitfalls of Amazon architecture and software limitations - however, everything was
sorted out and buzzed :-) We also carefully consider the issue of incremental backup of large enough data volumes (hundreds of gigabytes and more ), raids and configurations with InnoDB / XtraDB.
But first of all, we’ll take a look at the data storage technologies offered by Amazon.
Where do virtual machine data live?
So, let's start with what Amazon offers us for storing virtual machine data - virtual block devices EBS . Such a disk is easily created and connected to the server in 2 clicks. The maximum disk size is 1TB. By default, there is a limit on 5000 drives and 20TB, but it is increased at the first request .
The technology of local block devices is also proposed, the data on which ... disappear with the server (and this can easily happen when the machine crashes) - but I won’t write about it, because we did not experiment with her.
EBS Disk Performance
You can see right away that they are slower than the iron ones. Saturation of the device (% util, iostat command) with a random read volume of a dozen or two MB / s (even less by write) quickly approaches 100%. Slowing down is clearly seen on popular operations such as copying folders from disk to disk, unpacking archives, etc.
Raid?
In order to adequately start working with Amazon discs, the easiest way to “shove” them into a software raid. For databases, we use raid-10 on 8 EBS disks on both ext4 and xfs. A software raid is done quite simply , it works for a long time and practically does not break.
A raid can be especially useful if the EBS drive suddenly crashes.
Nevertheless, for some tasks we do not use raids - for example, for storing the MySQL binary log, for backups, etc. And for storing the nginx cache, raid0 on EBS disks worked well, which has been working steadily without failures for about a year.
EBS Drive Reliability
To be honest, for a year and a half of working with Amazon EBS-disks, they never let us down (no stupid things like “bads”, reading errors, etc.) ... except for the case when lightning hit the Irish data center, then several disks flew at once from raid-10 :-)
However, if you carefully read what Amazon writes about the reliability of their disks, you understand that you also need to do the raid and, of course, regular backups:
Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% - 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.
On the other hand, we have more than a hundred loaded EBS disks in production and in a year and a half, software raids have never knocked out disks due to IO errors. I’m sure we would have changed more than one device on the “iron” drives, so draw conclusions.
Available backup technologies
When the data is relatively small and they do not change often, you can play around with tar. But imagine a large online store that stores business information both in the database and in files on disk: new files appear every minute, and the total content size is hundreds of gigabytes.
DRBD? Yes, but they did not try this technology in Amazon, and I often hear from colleagues about its incredible braking when errors occur.
LVM and snepshoty mode copy-on-write - the similarity of this technology, only with extra buns and offers us the Amazon. Snapshots of a block device can be done as many times as necessary. Wherein:
- ONLY CHANGES get into the next snapshot of the EBS disk. Moreover, it is completely transparent and becomes obvious when you look at the monthly bill on the use of disk space. Even if you have 100 snapshots from a 500GB disk, but the data did not change often - you pay for about 500-500GB, which of course plays in favor of the client.
- You can and should remove snapshots - to maintain a balance between the size of the backup window and the cost of data storage. At the same time, which causes delight, you can delete ANY snapshot - Amazon automatically consolidates the data in the right way. And it doesn’t hurt my head on the topic - where is the basic snapshot, and where is the incremental snapshot - it doesn’t matter, you remove superfluous from any position (those who worked with Acronis will appreciate the convenience).
- Snapshots of disks are saved in S3 . S3 - as everyone probably already knows, this is a repository of objects of any format that replicates data to at least another 2 data centers. Those. the disk snapshot becomes almost “indestructible” and is stored more reliably than the hard drive in a locked nightstand under the desktop :-).
- A snapshot of the disk is done almost instantly - then in the background the data is transferred to S3 for a certain time (sometimes tens of minutes - if the Amazon is loaded).
All this means that we can take snapshots of a huge folder of frequently changing content on the disk at least once every 5 minutes - they will be stored securely in S3 and if you need to roll back 1TB of mutable data 5 minutes ago - we can do this with ease:
- Create a disk from a saved snapshot.
- We connect a disk to the server.
Of course, it is technically impossible to instantly transfer 1TB of data from S3 to the SAN where EBS disks live, therefore, although the block device becomes available to the operating system, data will be poured onto it in the background for a certain time - therefore, the speed of working with the disk at first will not be very high. But, nevertheless, you must agree how convenient it is to make an incremental backup of a large amount of data and roll it back to any point, for example, a week ago with a step of 5 minutes? :-)
In addition to the ability to create snapshots from EBS-drives, you can send files to S3 directly. Easy to use utility s3cmd- you can synchronize the filesystem trees with the cloud in both directions (only changes are transferred based on the calculation of md5 on the local disk and storage of the md5 object inside s3 in “ETag”). We tried solutions based on FUSE - s3fs technology , but noticed slowdowns and long-term freezes with the growth of LA during its intensive use.
Snapshot Raid
As I wrote above, EBS disks show adequate performance if they are combined into raid0, raid10. How to backup a raid? Snapshot each drive in turn? :-) We understand that it is impossible and the Amazon here does not offer us anything.
Good people have written a handy utility - ec2-consistent-snapshot . You can use it, or you can repeat its logic in scripts.
- We use a file system that supports "freezing" - i.e. realizing that she is now taking a snapshot at the block device level and it is necessary to flush buffers, commit transactions and temporarily stop block changes. Until recently, XFS understood this command ( xfs_freeze ), but in the “latest” linux distributions it became possible to “freeze” other common file systems: ext3 / ext4, xfs, jfs, reiserfs.
- We discard the changes and briefly prohibit writing to the FS: “fsfreeze -f mountpoint”
- Making snapshots of each raid disk: AWS API call CreateSnapshot .
- Allow the FS entry: “fsfreeze -u mountpoint”
If you have xfs, you can use the xfs_freeze command .
To connect a saved raid, it is better to write a script that connects the disks to the machine and launches a software raid from them. The raid saved in snapshots by the above method rises beautifully without losing the file system log - we use it in different places in production.
So, we learned how to snapshot raids in s3 with any amount of data with a frequency of at least once every 5 minutes and restore data from them. I’m sure that many of these things will come in handy on various projects in the cloud.
Whole machine snapshot
Sometimes it’s more convenient not to bathe separately with each raid, but to make a snapshot of all the disks of the machine with one command . You can make a snapshot in 2 modes: with a car stop and without a stop. In the latter case, we are logically warned about a possible “corruption” of data on disks / raids:
When taking a snapshot of a file system, we recommend unmounting it first. This ensures the file system metadata is in a consistent state, that the 'mounted indicator' is cleared, and that all applications using that file system are stopped and in a consistent state. Some file systems, such as xfs, can freeze and unfreeze activity so a snapshot can be made without unmounting.
After creating a machine snapshot, an AMI (Amazon Machine Image) object appears, with links to the saved snapshots of each drive. You can run a server with all disks / raids from this object with one command - AWS API call RunInstances . Did you feel the power of technology? Working servers can not only be backed up entirely, but also lifted from the backup TOTAL with all raids with one team! This technology saved us dozens of hours of system administration during the Amazon accident in August last year - we took the machines out of snapshots and deployed the configuration in another data center.
However, there is a serious pitfall - the CreateImage command is completely opaque and it is unclear how long it takes snapshots from all server disks - a second or 10 seconds? The method of scientific poking was chosen interval - 5 seconds, allowing you to shoot complete images of the machine with raids. I warn you - thoroughly test the script before launching into production - however, before the “goodies” of the technology for creating a full backup of the machine, you must admit that it’s hard to resist :-)
MySQL incremental backup
Let me remind you of our task - to backup a project with traffic of millions of hits per day and hundreds of gigabytes of content that changes quite often (the heaviest content has been moved to s3 and downloaded separately). I will repeat the known reasonable approaches to MySQL backup:
- Logical backup with slave. In this case, we do not slow down the operation of the main server, however ... we risk backing up “out of sync” data (so you need to monitor synchronization, for example using pt-table-checksum ).
- Binary snapshot using LVM from a combat server / slave, or - copy the blocks to a DRBD disk on the backup machine.
- Incremental binary backup from a combat server or slave using xtrabackup or a similar paid tool .
To be able to quickly roll back a large online store 5-10 minutes ago in the event of a catastrophic deletion of data in the database (an erroneous request that kills data in several order tables - who else was not with? :-)) - it seems that only 3 option will work . However, as it turned out, the binary incremental backup upon creation exerts a considerable load on the already weak EBS disks, but it can take several hours to restore increments to the basic binary backup during recovery ...!
I do not consider here recovery scenarios from a logical backup with preliminary editing of the MySQL binary log - it is still not fast to do this.
And here the Amazon helps us again. MySQL incremental backup is done like this:
- Flush MySQL / InnoDB / XtraDB buffers to disk: “FLUSH TABLES WITH READ LOCK”
- We discard the changes and briefly prohibit writing to the FS: “fsfreeze -f mountpoint”
- We make a snapshot of all disks of the machine: CreateImage . See above for pitfalls. If there are concerns, we make snapshots of each raid disk from the database: AWS API call CreateSnapshot .
- Allow the FS entry: “fsfreeze -u mountpoint”
- We remove the global locking of all tables in all databases: “UNLOCK TABLES”.
Now we have an AMI object with a hot MySQL backup and we have made the maximum possible to start it from the backup as quickly as possible.
Thus, it turned out to be pleasant just to make an incremental backup of the MySQL server in S3 with a frequency of at least once every 5 minutes and the possibility of its quick entry into production. If the server is used in replication, then it will restore it as a rule without any problems, if you did not forget to set conservative replication settings in the settings (well, or you can return it to work pretty quickly manually):
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1
sync_master_info = 1
sync_relay_log = 1
sync_relay_log_info = 1
How to script actions with Amazon?
For the system administrator, there are convenient utilities that pull the Amazon REST methods. Several utilities are downloaded for each web service used, and calls to utilities are scripted in bash. Here is an example of a script that changes hardware on a server:
#!/bin/bash
#Change cluster node hw type
#Which node to change hardware?
NODE_INSTANCE_ID=$1
#To which hw-type to change?
#Some 64-bit hw types: t1.micro (1 core, 613M), m1.large (2 cores, 7.5G), m1.xlarge (4 cores, 15G),
#m2.xlarge (2 cores, 17G), c1.xlarge (8 cores, 7G)
NODE_TARGET_TYPE='c1.xlarge'
#To which reserved elastic ip to bind node?
NODE_ELASTIC_IP=$2
ec2-stop-instances $NODE_INSTANCE_ID
while ec2-describe-instances $NODE_INSTANCE_ID | grep -q stopping
do
sleep 5
echo 'Waiting'
done
ec2-modify-instance-attribute --instance-type $NODE_TARGET_TYPE $NODE_INSTANCE_ID
ec2-start-instances $NODE_INSTANCE_ID
ec2-associate-address $NODE_ELASTIC_IP -i $NODE_INSTANCE_ID
For developers, there are libraries in different languages for working with the Amazon API. Here is the library for working from PHP - AWS SDK for PHP .
As you can see, scripting work with Amazon objects is simple.
Summary
Using concrete examples diluted with the theoretical base of popular practices and salted with effective Amazon API tools, we learned:
- incrementally backup and restore large amounts of data from EBS disks and Amazon raids in S3 and vice versa
- backup and restore entire Amazon machines
- incrementally backup and quickly put into operation a MySQL server with data stored on raids of EBS disks
- discussed several pitfalls
- looked how easy it is to script the main actions
In one of the following articles I will tell you how to effectively manage traffic and balance the load between clusters and data centers of Amazon.
In practice, we actively use the technologies listed in the article, we experiment, are open and ready to share our experience and look for simple and effective solutions to emerging problems using the tools of the open source world.
I invite everyone to register on our Bitrix24 cloud service and see what happened. I also invite everyone to the FREE seminar on web clusters and high loads, which will be held in the conference room of 1C on May 22.
Good luck to everyone in building scalable and fault-tolerant systems!
