Backups through bacula on Amazon S3

As you know, all people are divided into two types: those who still do not make backups, and those who already do them. For those who are just starting to make backups, the first question that usually arises is how to archive data. We will not consider simple options (manually cut the blanks, completely archive directories to other servers) - they have very modest capabilities for indexing and searching archive files. Instead, we turn to automatic backup systems, in particular bacula. This article does not address the question of why bacula. The main reasons - it is distributed under a free license, is available for a bunch of platforms and has great flexibility.

The second question after choosing an archiving system is choosing a place to store backups. Bacula allows you to use streamers, CDs, write archives to FIFO devices and to regular files. The streamer is convenient on corporate servers, where there is a permanent physical to the hardware. Storage of archives in files is suitable when the volume of archives does not exceed the volume of hard disks, plus for reliability of storage it is desirable to make a RAID array with redundancy, or even several physical servers for backups, preferably in different rooms. Otherwise, all this until the first fire. Cutting into discs is a home option, the main disadvantage of which is the need to regularly stick fresh discs. We configured bacula to archive data on Amazon S3.

Amazon S3 is Amazon’s cloud-based file storage that is reasonably priced and suitable for archiving any servers that are constantly connected to the Internet. This can be home computers, and office servers, and servers on the sites of data centers.

So, let's get started with the setup. An office local network and a small cluster in a data center are archived at our farm.

Director

Bacula is designed in such a way that the central server, Director, plays a central role in it. Its function is to store information about the entire archiving system, according to the schedule, start the necessary tasks and log the results of their implementation. On the office network server, install the bacula-director-mysql package (as it is called in Debian).

Bacula has excellent configuration documentation. I will dwell only on key settings. The Catalog section indicates the address of the MySQL server where the archive index, database name, name and password will be stored. In the Messages section, we write the administrator’s e-mail, where reports on the results of execution will be dumped:

mailcommand = "/usr/lib/bacula/bsmtp -h smtp.example.com -f \"\(Bacula\) %r\" -s \"Bacula daemon message\" %r"
mail = [email protected] = all, !skipped

Storage

Bacula has a distributed architecture. The servers to which the storage media is connected to which archives will be recorded are called Storage. It is Storage servers that will interact with Amazon S3. In our system, we needed two Storage servers - one is installed in the office, directly on the server with Director (all data from office computers will be backed up to it), the other on one of the servers in the data center (data from other computers in the data center will go there). The meaning of separation is not to drive paid traffic between the data center and the office.

By registering for Amazon AWS and creating a bucket with your chosen name, you will have the following details on hand: bucket name, access_key and secret_key. The next step is to use the s3fs module for fuse to mount our bucket directly into the server file system. s3fs is not part of the Debian distribution - you will need to build it manually. For some reason, the latest versions didn’t work right away (there were some kind of freezes after deleting files, and you had to remount the file system), but version 1.16 worked right away without a single problem. We write in the server boot:

s3fs your-bucket-name /mnt/backup -o allow_other,retries=10,connect_timeout=30,readwrite_timeout=30

And create the file / etc / passwd-s3fs with S3 passwords:

access_key:secret_key

After the file system is mounted, we install the bacula-sd package and ask bacula to save the archive files in S3. To do this, in the bacula-sd.conf configuration file in the Storage section, specify Name so that we can easily identify it, for example Name = companyname-office-sd, Maximum Concurrent Jobs = 1 so that there are no simultaneous access to different files through S3 . In the Director section, write the name of our Director server and some random password. In the Storage section, we describe Device - the physical location for storing archives:

Device {
    Name=S3-companyname-office
    Media Type=file
    ArchiveDevice=/mnt/backup/office
    Label Media=yes;
    Random Access=yes;
    AutomaticMount=yes;
    RemovableMedia=no;
    AlwaysOpen=no;
}

now for a moment we’ll go back to the Director server and create a new Storage section in its config:

Storage {
        Name = S3-companyname-office
        Address = storage.server.domain.name
        SDPort = 9103
        Password = "storage-server-password"
        Device = S3-companyname-office
        Media Type = File
        Maximum Concurrent Jobs = 1
}

Password is the same as in the Storage config in the Director section.

File daemon

File Daemon is actually a daemon installed on computers that need to be backed up. The bacula-fd package is installed on all computers in the cluster, on office computers with valuable data, and its bacula-fd.conf config is configured. In the Director section, we write the identifier of our Director server and come up with a new random password. On this, its configuration ends, and we again return to the Director server to register a new client.

In the Director config, create a new section:

Client {
        Name = server-name-fd
        Address = this.server.host.name
        FDPort = 9102
        Catalog = MyCatalog
        Password = "file-server-password"
        Maximum Concurrent Jobs = 1
        AutoPrune = yes
        Job Retention = 365 days
}

Password is the same as in the File Daemon config in the Director section.

An important parameter is Job Retention. After this time, old data will be deleted from the archive index. The longer this time, the longer your old archive files will not be overwritten, and the more money you will pay for Amazon S3. The smaller, the cheaper backups will cost you, but the depth of archiving will also be less. In addition, make sure that you have full (Full) archiving more often than Job Retention, otherwise you will have moments when the old data has already been deleted and the new ones have not yet been archived.

Free up space

Please note that physically the data is not deleted from the archive, even if the links to it are deleted from the index after the Retention interval has expired. Removing from the index simply means that new files can be written in place of the old ones. The disk space is not freed up, and you won’t pay less, even if you clear the entire index. Physically, you can either delete files manually, making sure that there are no links left in the index, or install the latest version of bacula, which automatically can truncate freed volumes. In practice, this is rarely needed, and we do not use this opportunity.

Firewall Settings

During operation, bacula requires three types of TCP connections: from Director to Storage on port 9103, from Director to File Daemon on port 9102 and from File Daemon to Storage on port 9103. In all these directions, firewalls should be open, and the addresses that prescribed in the Address parameters on Storage and File Daemon, should be available in the specified directions. In particular, if you suddenly have Storage inside the local network, then Director and File Daemons, which will be archived to it, must also be inside the same network. If suddenly for some reason you need to keep Storage inside the network, then you need to configure routing so that the corresponding port is forwarded to the inside of the network.

Features s3fs

When writing to a file, s3fs reads the entire previous version of the file onto the computer, all modifications occur in the local copy, and after closing the file, it is completely uploaded to S3. This means that even adding a few bytes to an archive file of 500 megabytes in size will lead to the transfer of a gigabyte over the network. Since traffic to Amazon S3 is charged, you should not forget about this feature of s3fs. The file size should not be made very large so that downloads and uploads are small. We have 3 pools on each Storage server - for Full backups (Maximum Volume Bytes = 500000000, Volume Retention = 12 months), for Diff backups (Maximum Volume Bytes = 300000000, Volume Retention = 7 months) and for Incremental -backups (Maximum Volume Bytes = 100000000, Volume Retention = 2 months). Note, that in this example, differential backups should be done at least every 2 months (we have 1 month), otherwise there will be times when incremental backups are already deleted, and there is no fresh differential backup. Similarly, full backups should be done at least once every 7 months (we have 6).

In addition, since you need a place to store local copies, you need to always have it on the file system. Where to add local copies is specified in the use_cache option when mounting s3fs. By default, this is the root FS. The more concurrent files you have open, the more space there will be required. Therefore, we limit both Maximum Concurrent Jobs on one Storage server (so that many files are not kept open), and Maximum Volume Bytes. If the place suddenly runs out, s3fs will freeze and wait for the place to free up.

Issue price

Storage prices range from $ 0.037 per gigabyte per month (for large storage volumes with reduced reliability of 99.99%) to $ 0.14 per gigabyte per month (for small storage volumes with standard reliability of 99.999999999%). We chose standard reliability and store about a terabyte of archives - this costs (together with the cost of traffic) about $ 180 per month.

Security

Here's a short security checklist:

• block read access for everyone to the bacula configs and to the / etc / passwd-s3fs file, because disclosing passwords recorded there is extremely dangerous;
• restrict access to ports 9101, 9102 and 9103, with the exception of the directions necessary for bacula operation;
• configure bacula to use TLS when transmitting data over public networks;
• close catalogs with archives from prying eyes;
• fuse encryption means encrypting data that will reach Amazon S3;
• to different S3 buckets on different Storage-servers, so that in case of compromise of one archive, the hacker does not gain access to others;
• Perform test restorations regularly to make sure you remember to put something important in the archives.

References

• Bacula - www.bacula.org
• Amazon S3 - aws.amazon.com/s3
• s3fs - code.google.com/p/s3fs/wiki/FuseOverAmazon
• encryption in fuse - prefetch.net/blog/index.php/2007/05/29/encrypting-data-with-the-fuse-encryption-module