February 8, 2009 at Become a maintainer. Part four
While fans of exotic habret actively drink Java cups, take doses of F # and inject themselves with other Haskell, you and I learned to collect their creations in deb packages. Since some time has passed since the previous part, someone has probably already accumulated several ready-made packages, and so far we have not even tried to put them into the official Debian and Ubuntu repository. Therefore, it is time to think about how to organize all the accumulated wealth into one big beautiful repository, which will not be ashamed to offer other users for use.
(Parts 1 , 2, and 3 )
For these purposes, we will consider two programs (simply because I only needed them to comprehend the Tao) - this is apt-ftparchivefrom the standard apt-utils package and the reprepro package .
Let's first see how a normal repository works. Repositories are of two types - ugly and oblique (or in the original - flat - flat, we will not consider them) and normal hierarchical ones. Here we will learn to create hierarchical ones. So, the normal repository consists of two directories: pool and dists .
pool /
Packages may be in this directory in an arbitrary hierarchy. I say “arbitrary” because you can really lay them out as you like, but in practice the following hierarchy is usually implemented:
Here “main” is the name of the section to which we refer our package (see the first article for sections), then the first letter of the name of the src package and then the directory named after the src package. The src package and all binary packages collected from it will also lie in this directory. For numerous libraries where package names all begin with “lib”, instead of the letter “l”, a separate “liba” is used, where “a” is the first letter of the name after “lib”.
dists /
This directory is organized much more complicated. First of all, we should find a directory in it with the name of the repository - in ubuntu these are various “intrepid”, “intrepid-proposed”, “jaunty”, etc., in the debian it is “etch”, “etch-proposed-updates” , "Lenny" and their synonyms - "stable", "testing", etc. - they are usually symlinks to the canonical name. Thus, packages can be stored in one place for several repositories at once.
dists / unstable / [1]
Inside this directory you can find the following files:
In addition, in the same directory we will find directories with names corresponding to the names of the repository sections - main, contrib, non-free or main, universe, multiverse, restricted, or those that you yourself define.
dists / unstable / main /
In each section (not only main, of course), you can observe a set of directories of the following form:
dists / unstable / main / binary-i386 / The Packages
file(as well as Packages.gz and Packages.bz2 ) containing descriptions of binary packages of the corresponding architectureis stored here. In essence, it is a control-files from all binary packages of the repository connected for a given architecture. Often here you can still find the Release file, which, unlike the same file in dists / unstable /, does not contain file hashes, but only stores repository descriptions for this directory. dists / unstable / main / source / Here everything is the same as in binary-arch, except that the word Sources is used instead of the word Packages.
Of course, all these files can be written with pens, but I would not advise you. Especially if you have at least 3-4 packages. Therefore, we will use the magic utility apt-ftparchive from the apt-utils package and dpkg-scanpackages from the dpkg-dev package.
So, let's say we have a certain qutim src package that we created in the previous series, and the qutim and qutim-dev binary packages assembled from it. We collected them for two architectures, so now we have the following set of files:
The first three files, as we recall, are src-package, and the remaining are assembled binary packages. We also have qutim_0.1.99.138-1_i386.changes and qutim_0.1.99.138-1_amd64.changes files generated during package assembly.
Let's put together a minimally decent repository from them.
So, create a directory with the initial structure and copy the packages into it:
As you can see, there is nothing complicated in placing the packages themselves in the repository. It will be a little more difficult to generate a set of descriptions for these packages:
Here we generate descriptions for each architecture and for src packages, after which we put them in the corresponding file. Note that we generate three files each - Packages , Packages.gz and Packages.bz2 (similar to Source). In principle, this is not necessary, but we will be cultured people and we will generate all three - choose any one you like. The -a option indicates each time the packages for which architecture to choose from the whole set in pool / main. As I already said, the qutim-dev package will fall
Theoretically, and the description of the binary packages could be generated using apt-ftparchive, but I still haven’t found an option that selects packages of a specific architecture instead of shoving them into the output all.
Now we need to generate the Release file:
You still have to write the description with your hands, apt-ftparchive cannot do this.
And the last step is to sign our repository:
Voila, our repository can be used. Add the following lines to /etc/apt/sources.list:
Updating the list of packages - and you can use the repository.
Of course, this is not very convenient - you will have to regenerate the description files each time you update a set of packages. Of course, all this is automated by scripts, but still so uninteresting. Therefore, we will move on to the second program used, which will do everything for us.
This program is just a paradise for those who do not want to bother with the structure of the repository in general, but want to configure and use it once. Therefore, we demolish our homegrown repository and install the reprepro package:
Now we create a distributions file in the rep / conf directory, in it we will describe our repository:
Codename should contain the name of the repository (lenny, hardy, etch, jaunty), and Suite - a synonym for creating symbolic links (stable, unstable). The Version, Origin, Label and Description parameters are not particularly meaningful and are simply copied to the Release file. The SignWith parameter indicates the need to sign the repository by creating the Release.gpg file. DebIndices, DscIndices and Contents - indicate the need to create Packages, Sources and Contents-arch files - in a simple and archived form.
Now we can add our packages:
the -b switch tells reprepro where our repository is located, the --ask-passphrase command reports that our gpg-key is password-protected and it would be nice to ask for a password when signing the distribution, the -C switch tells us to include the package in the main section. One interesting point should be noted here. The include command takes as an argument the name of the repository into which to include the file and the name of the .changes file created during the assembly of the src package. At the same time, an src package and assembled binary packages for this architecture are added to the repository. And here we have a problem - reprepro developers flatly refuse to add packages to the repository of the version already existing in it. Therefore, if we now try to specify the .changes file for another architecture - we get an error - we have already added not only the src package, but also the qutim-dev package compiled for all architectures.
Browse the contents of the rep directory now. The conf subdirectory contains the repository configuration (more details can be found in ` man reprepro` ), the db directory contains its internal database. But the pool and dists directories became very different from what we created using apt-ftparchive and became similar to the structure that I described at the beginning. Since we already have the repository address in sources.list, type ` sudo apt-get update` and use it :)
Finally, if you want to study the structure of the repository in more detail, I recommend that you study the official Ubuntu repository . Those who wish, of course, can also get acquainted with the Debian repository , but I immediately warn that it is much more complicated.
* [1] hereinafter, instead of the abstract directory name, I will use specific examples
(Parts 1 , 2, and 3 )
For these purposes, we will consider two programs (simply because I only needed them to comprehend the Tao) - this is apt-ftparchivefrom the standard apt-utils package and the reprepro package .
Repository structure
Let's first see how a normal repository works. Repositories are of two types - ugly and oblique (or in the original - flat - flat, we will not consider them) and normal hierarchical ones. Here we will learn to create hierarchical ones. So, the normal repository consists of two directories: pool and dists .
pool /
Packages may be in this directory in an arbitrary hierarchy. I say “arbitrary” because you can really lay them out as you like, but in practice the following hierarchy is usually implemented:
pool / main / q / qutim /
Here “main” is the name of the section to which we refer our package (see the first article for sections), then the first letter of the name of the src package and then the directory named after the src package. The src package and all binary packages collected from it will also lie in this directory. For numerous libraries where package names all begin with “lib”, instead of the letter “l”, a separate “liba” is used, where “a” is the first letter of the name after “lib”.
dists /
This directory is organized much more complicated. First of all, we should find a directory in it with the name of the repository - in ubuntu these are various “intrepid”, “intrepid-proposed”, “jaunty”, etc., in the debian it is “etch”, “etch-proposed-updates” , "Lenny" and their synonyms - "stable", "testing", etc. - they are usually symlinks to the canonical name. Thus, packages can be stored in one place for several repositories at once.
dists / unstable / [1]
Inside this directory you can find the following files:
- Contents-arch - a file that contains a list of all files from all packages. Instead of “arch”, the name of the package architecture is substituted, and at the end there may also be “.gz” and “.bz2” (if the file is compressed accordingly). It is optional and is used mainly by utilities such as apt-file to find in which package the required file can be found. Example name - Contents-amd64.gz
- Release is a file that contains a brief description of the repository and hashes of all supporting files contained in dists / .
- Release.gpg is the gpg signature of the Release file, certifying the creation of the file by a specific author. Both files are optional, but in their absence APT is guaranteed to swear at an unauthenticated repository.
In addition, in the same directory we will find directories with names corresponding to the names of the repository sections - main, contrib, non-free or main, universe, multiverse, restricted, or those that you yourself define.
dists / unstable / main /
In each section (not only main, of course), you can observe a set of directories of the following form:
- binary-arch - a directory for describing packages of the corresponding architecture, for example, binary-amd64 - for the architecture of amd64. All architecture packages (such as dev packages) will be mentioned in the directory for each architecture.
- source - src packages are described in this directory
- i18n - this directory is rarely used and mainly only in official repositories - it contains files with translation of package descriptions into other languages. I will not describe this directory, because it is not such a really important element, if you want to use it, you can always look in the repository ubuntu
dists / unstable / main / binary-i386 / The Packages
file(as well as Packages.gz and Packages.bz2 ) containing descriptions of binary packages of the corresponding architectureis stored here. In essence, it is a control-files from all binary packages of the repository connected for a given architecture. Often here you can still find the Release file, which, unlike the same file in dists / unstable /, does not contain file hashes, but only stores repository descriptions for this directory. dists / unstable / main / source / Here everything is the same as in binary-arch, except that the word Sources is used instead of the word Packages.
apt-ftparchive
Of course, all these files can be written with pens, but I would not advise you. Especially if you have at least 3-4 packages. Therefore, we will use the magic utility apt-ftparchive from the apt-utils package and dpkg-scanpackages from the dpkg-dev package.
So, let's say we have a certain qutim src package that we created in the previous series, and the qutim and qutim-dev binary packages assembled from it. We collected them for two architectures, so now we have the following set of files:
qutim_0.1.99.138.orig.tar.gz
qutim_0.1.99.138-1.diff.gz
qutim_0.1.99.138-1.dsc
qutim_0.1.99.138-1_amd64.deb
qutim_0.1.99.138-1_i386.deb
qutim-dev_0.1.99.138-1_all.debThe first three files, as we recall, are src-package, and the remaining are assembled binary packages. We also have qutim_0.1.99.138-1_i386.changes and qutim_0.1.99.138-1_amd64.changes files generated during package assembly.
Let's put together a minimally decent repository from them.
So, create a directory with the initial structure and copy the packages into it:
$ mkdir -p rep/dists
$ mkdir -p rep/pool/main
$ cp qutim_0.1.99.138.orig.tar.gz qutim_0.1.99.138-1.diff.gz qutim_0.1.99.138-1.dsc qutim_0.1.99.138-1_amd64.deb qutim_0.1.99.138-1_i386.deb qutim-dev_0.1.99.138-1_all.deb rep/pool/main/
$ cd repAs you can see, there is nothing complicated in placing the packages themselves in the repository. It will be a little more difficult to generate a set of descriptions for these packages:
$ mkdir -p dists/unstable/main/binary-amd64
$ mkdir dists/unstable/main/binary-i386
$ mkdir dists/unstable/main/source
$ dpkg-scanpackages -a amd64 pool/main >dists/unstable/main/binary-amd64/Packages 2>/dev/null
$ dpkg-scanpackages -a amd64 pool/main | gzip -c9 >dists/unstable/main/binary-amd64/Packages.gz 2>/dev/null
$ dpkg-scanpackages -a amd64 pool/main | bzip2 -c9 >dists/unstable/main/binary-amd64/Packages.bz2 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main >dists/unstable/main/binary-i386/Packages 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main | gzip -c9 > dists/unstable/main/binary-i386/Packages.gz 2>/dev/null
$ dpkg-scanpackages -a i386 pool/main | bzip2 -c9 > dists/unstable/main/binary-i386/Packages.bz2 2>/dev/null
$ apt-ftparchive sources pool > dists/unstable/main/source/Sources 2>/dev/null
$ apt-ftparchive sources pool | gzip -c9 > dists/unstable/main/source/Sources.gz 2>/dev/null
$ apt-ftparchive sources pool | bzip2 -c9 > dists/unstable/main/source/Sources.bz2 2>/dev/nullHere we generate descriptions for each architecture and for src packages, after which we put them in the corresponding file. Note that we generate three files each - Packages , Packages.gz and Packages.bz2 (similar to Source). In principle, this is not necessary, but we will be cultured people and we will generate all three - choose any one you like. The -a option indicates each time the packages for which architecture to choose from the whole set in pool / main. As I already said, the qutim-dev package will fall
Theoretically, and the description of the binary packages could be generated using apt-ftparchive, but I still haven’t found an option that selects packages of a specific architecture instead of shoving them into the output all.
Now we need to generate the Release file:
$ echo "Archive: unstable" > dists/unstable/Release
$ echo "Suite: unstable" >> dists/unstable/Release
$ echo "Components: main" >> dists/unstable/Release
$ echo "Origin: qutim.org" >> dists/unstable/Release
$ echo "Label: qutim.org Debian repository" >> dists/unstable/Release
$ echo "Architectures: amd64 i386" >> dists/unstable/Release
$ echo "Description: Debian qutIM unstable" >> dists/unstable/Release
$ apt-ftparchive release dists/unstable >> dists/unstable/ReleaseYou still have to write the description with your hands, apt-ftparchive cannot do this.
And the last step is to sign our repository:
$ gpg -abs -o dists/unstable/Release.gpg dists/unstable/ReleaseVoila, our repository can be used. Add the following lines to /etc/apt/sources.list:
deb file: /// path / to / rep unstable main
deb-src file: /// path / to / rep unstable main
Updating the list of packages - and you can use the repository.
Of course, this is not very convenient - you will have to regenerate the description files each time you update a set of packages. Of course, all this is automated by scripts, but still so uninteresting. Therefore, we will move on to the second program used, which will do everything for us.
reprepro
This program is just a paradise for those who do not want to bother with the structure of the repository in general, but want to configure and use it once. Therefore, we demolish our homegrown repository and install the reprepro package:
$ cd .. && rm -rf rep/*
$ sudo apt-get install reprepro
$ mkdir rep/confNow we create a distributions file in the rep / conf directory, in it we will describe our repository:
Codename: lenny
Suite: unstable
Version: unstable
Origin: qutim.org
Label: qutim.org Debian Repository
Description: qutim.org Debian repository
Architectures: source i386 amd64
Components: main
SignWith: default
DebIndices: Packages Release. .gz .bz2
DscIndices: Sources Release. .gz .bz2
Contents:. .gz .bz2
Codename should contain the name of the repository (lenny, hardy, etch, jaunty), and Suite - a synonym for creating symbolic links (stable, unstable). The Version, Origin, Label and Description parameters are not particularly meaningful and are simply copied to the Release file. The SignWith parameter indicates the need to sign the repository by creating the Release.gpg file. DebIndices, DscIndices and Contents - indicate the need to create Packages, Sources and Contents-arch files - in a simple and archived form.
Now we can add our packages:
$ reprepro -b rep/ createsymlinks
$ reprepro -b rep/ --ask-passphrase -C main include unstable qutim_0.1.99.138-1_amd64.changes
$ reprepro -b rep/ --ask-passphrase -C main includedeb unstable qutim_0.1.99.138-1_i386.debthe -b switch tells reprepro where our repository is located, the --ask-passphrase command reports that our gpg-key is password-protected and it would be nice to ask for a password when signing the distribution, the -C switch tells us to include the package in the main section. One interesting point should be noted here. The include command takes as an argument the name of the repository into which to include the file and the name of the .changes file created during the assembly of the src package. At the same time, an src package and assembled binary packages for this architecture are added to the repository. And here we have a problem - reprepro developers flatly refuse to add packages to the repository of the version already existing in it. Therefore, if we now try to specify the .changes file for another architecture - we get an error - we have already added not only the src package, but also the qutim-dev package compiled for all architectures.
Browse the contents of the rep directory now. The conf subdirectory contains the repository configuration (more details can be found in ` man reprepro` ), the db directory contains its internal database. But the pool and dists directories became very different from what we created using apt-ftparchive and became similar to the structure that I described at the beginning. Since we already have the repository address in sources.list, type ` sudo apt-get update` and use it :)
Finally, if you want to study the structure of the repository in more detail, I recommend that you study the official Ubuntu repository . Those who wish, of course, can also get acquainted with the Debian repository , but I immediately warn that it is much more complicated.
* [1] hereinafter, instead of the abstract directory name, I will use specific examples