Aptly - creating your own repository

My organization writes software for Linux.
The software is designed to work at retail outlets that are geographically distributed.
Initially, the software was provided to clients as a set of deb packages for different distributions and architectures with a list of packages that need to be installed as dependencies before installing these deb files.
I want to tell you what the evolutionary path was from distributing files via ftp to creating a repository and launching a configuration management system and starting implementing a continuous integration server.

The origin of life


So, software is a commercial development written in Qt, everything is built for two architectures (i386 and amd64) and for several distributions. At the moment, we decided this: two or three recent releases of Debian and the last two LTS from Ubuntu. Plus, there are several versions (currently three) that are used by clients.
Three versions turned out this way: when buying software, the client is offered support and, for reasonably reasonable money, the provision of new versions of software. Changing minor versions is provided free of charge, regardless of the availability of support. Major - or with support, or sold again only at a discount.

While there were few versions, and the number of installations for clients was small, ftp could be quite costly. It looked like this: after assembling all the sources in deb packages, the set of files for each client (depending on the version) was archived and laid out for each client on ftp in its own section. Over time, ftp accumulated a lot of identical tar-s from different clients, which had to be removed from time to time.
As time went on, clients were added, client networks grew in size, and updating the software version on points became another task, especially considering that at 99% of points the Internet was (and still was) GPRS.

Speaking of collecting packages. The skips that collected deb packages were originally written “300 years ago” on some manual in English and IMHO is not entirely correct. The following articles helped to comb and wash the scripts (for which I want to say thanks to their authors):

The beginning of evolution


The question of distributing compiled packages was somehow raised in some more correct way. “And then Ostap suffered” (c), decided to do everything right, so that a minimum of time was spent on routine operations.

Reprepro was chosen as the software that supported the repository. At that time, the opportunities that he provided were quite enough.

Further, based on the experience of updating, configuring a large number of computers, I wanted to have a configuration management tool. Google and Habr were called in to help. I got acquainted with ansible, chef, puppet and other systems whose names I no longer remember. According to the understandability of the configs, the documentation, the entry threshold and the combination of other parameters, puppet was adopted. The foreman was bolted to the puppet. And happiness has come.

Not much time passed and I wanted something more different as a repository management. Reprepro suited everyone, except for the ability to store different versions of the same package in the same repository: to be able to quickly roll back to a specific version in case of some kind of regression, or to quickly put the right version and test the behavior of the system of a certain version on a specific data set on certain hardware configurations.

As an example: one client detected an application crash while working with documents. Their data set on the computers of the developers flew without question, everything opened and did not fall wonderfully. We put together a stand, installed the same linux, the same version of software, again everything works and does not crash. They started to act as paranoid as possible - they assembled another machine as close as possible to the client’s machine, the amount of disk, memory, processor, monitor resolution and lo and behold! repeated the fall. We began to understand: we found out that the essence is in the resolution of the monitor, or rather, not in the resolution as such, but in the aspect ratio. The developers and on the first test machine, the aspect ratio of the monitor was 16: 9, and the second test 4: 3. As a result, we calculated that one line in qss leads to a fall. Which was, of course, a big surprise. So this is what this example is for. At the time of these tests, there was no such repository, and for each assembled machine we had to copy deb packages through scp in the old fashioned way, add dependencies with pens, then install the packages themselves and only then check them. Instead of installing in a couple of minutes through aptitude with all the dependencies.

I had to google. As a result, the article was googled . I began to deal with the options described in it. I managed to look at:
  • DAK - not mastered at all. I started exactly with it as an “official solution”, but to my shame I couldn’t understand what to do with the project tree that git pulled to me. Therefore, he did not bother much and continued to read further.
  • The second product that I noticed was mini-dak. And as if everything is clear from the config and the description is more or less clear. I just stopped the inability to make repository names (what in terms of mini-dak is called suite) such as I want. For example, I want to get the path to the repository like this:
    deb http: // SERVER_NAME / squeeze-evolution-beta non-free
    A can’t, because somewhere in the depths of the scripts a variable name is created with the signs “-” which does not work in bash-e. Attempts to modify the scripts with a file were unsuccessful, because my level of programming on bash does not reach the level of the person who wrote them, but I did not want to kill a lot of time studying all the intricacies of bash (though not a bad pastime).
  • The third option for review was aptly selected. And it seems like this tool is suitable.

Work with aptly


The documentation for aptly is quite detailed and with examples. In addition there is bash-completion.
The possibilities of aptly are very wide, below I will give only those commands that I use. If you are interested, I recommend the product website more deeply .

Creating a repository:

You can create immediately with all the parameters:
aptly repo create -comment="Wheezy prehistoric" -distribution="wheezy-prehistoric" -architectures="i386,amd64,all" -component="non-free" wheezy-prehistoric

You can add parameters later, and create just like that:
aptly repo create wheezy-prehistoric

Change settings

Change the parameters one at a time:
aptly repo edit -comment="Wheezy prehistoric" wheezy-prehistoric

Or all at once:
aptly repo edit -comment="Wheezy prehistoric" -distribution="wheezy-prehistoric" -architectures="i386,amd64,all" -component="non-free" wheezy-prehistoric

View existing repositories

aptly list

Retrieving repository information

general information:
aptly show wheezy-prehistoric

package list information:
aptly repo show -with-packages wheezy-prehistoric

Adding packages to the repository

single file:
aptly repo add wheezy-prehistoric build//Debian-wheezy/chameleon-core_1.3.0-wheezy46_amd64.deb

entire catalog:
aptly repo add wheezy-prehistoric build//Debian-wheezy

when adding, I was somewhat confused by the lack of autocompletion of the path to the file / directory

Repository publishing

aptly publish repo wheezy-prehistoric

Delete publication

aptly publish drop wheezy-prehistoric

Get repository graph

aptly graph


There is also support for snapshots, creating mirrors, moving packages between repositories, dependency support, an integrated HTTP server for working with published repositories.

There is one peculiarity about which I wrote to the author - pool is one common for all repositories, which is correct, only if I have files with the same name in different repositories, but there are no messages with different contents and sizes when publishing these repositories, logically had at least something to write. And in the pool remains the file of the first published repository.
By the way, the author reacted very quickly and created a task on the bug tracker .

Evolution continues


At the moment, the packages are being collected by handles on vitro machines. What is not good. Already begun work on the introduction of Jenkins . It was he who was selected from the entire list of available tools for CI. TeamCity was originally installed, but the cost of licenses was not understood due to the fact that there are other projects that are completely free and have no worse functionality. At least it looks that way now. If during the operation of Jenkins, its capabilities for some reason will not be enough (although with so many plugins it is hardly believed), we will change for something better.

PS


At the end of the article I would like to summarize.
The idea of ​​the article was to describe a wonderful tool - aptly, because on open spaces Habr did not find anything about him. Well, to say in the style of “here’s the tool - you can use it”, I found it not interesting. I decided that it would be much more interesting to read which way my organization developed and which tools I used at the same time and draw the attention of the community to a useful tool using a personal example.

Thanks for attention.

Also popular now: