On some aspects of organizing a UNIX / Linux file system

1.0 Introduction

After writing the previous article ( Linux: Installing programs not included in the distribution using the xstow manager ), I had a double impression. On the one hand, everything is correct in the article, and on the other hand, the reviews showed that there are some discrepancies in the designation of the various parts of the UNIX file system. It so happened that I gave people a hammer in my hands, gave instructions for using the hammer, and did not say which nails and where to hammer with this hammer. I will try to fill this gap. In this article I will try, as far as I can, to tell how the UNIX file system is organized, why this is done exactly this way, why and how to behave in this system.



In the future, I will use the Debian / Ubuntu distribution as examples of UNIX.

As we know from life, our whole life is a solution to the problems that arise before us, the emergence of new problems, including those arising from the solution of previous ones and so on, to infinity ...

2.0 Problem: how to place files for various purposes in the OS

So let's formulate the first problem, how to place files in the system? On the one hand, we have an operating system and programs, each of which usually consists of several, sometimes many files. If everything is more or less clear with the basic files of the operating system, as a rule they should always be in the same place, then with the files of programs for which installation / uninstallation is possible, the situation is more complicated. On the one hand, it’s good when the files are sorted by purpose, on the other hand, it’s convenient when the directory trees of the program files are in directories corresponding to the programs. Let's look at the pros and cons of these solutions in more detail.

2.1 First option: sort files by packages

The first thing that comes to mind, let's decompose the OS files into packages. Those. for each software package there will be its own directory in which there will be subdirectories in which the files will already be laid out as the manufacturer of this program needs it.

Pros:

• Each manufacturer puts the files in his directory as he needs.
• It is easy to install / uninstall programs, copy or destroy the directory, and all things.

Minuses:

• The operating system searches for executable files or shared libraries from a list that is in the corresponding environment variable. For example, paths to executable files are listed in the $ PATH environment variable . When there are a lot of packages, these lists become really huge.
• There is another good reason why it is desirable to place files of similar purpose for different purposes in the same places. The fact is that for various reasons, the properties of directories / file systems for files for various purposes should be different. This is necessary, sometimes for security reasons, sometimes for reasons of optimal system configuration. UNIX has the ability to mount file systems in directories. What does it give? You can, for example, mount the directory for the binaries to be mounted in read only mode, and then no malicious program can write anything to it, and mount the other directory in the mode that prohibits launching, and then no one can start anything from this FS. Files that must be changed (for example, database files) are located on another file system,

2.2 Second option: sort files by functions in the system

Pros:

• You can optimally configure file systems based on file functions.
• You can optimally configure permissions on files and directories and mounted file systems.
• It is possible and convenient to give access to such directories over the network.

Minuses:

• There are a lot of files in shared directories, for which it is not clear which package they belong to. There is a problem removing packages.

3.0 Solution in UNIX-style

In UNIX, the second option is selected. Those. files of different programs with the same functionality are in the same directory. For example, the executables of various user programs are in the / usr / bin directory .

4.0 Problems arising from a solution in UNIX-style, software installation

Problems, of course, arise from the downsides of the approach. These are difficulties with installing programs, uninstalling programs, and determining which file belongs to which program. This is almost always a non-trivial task.

5.0 Problem Solving: Package Managers

In the future, we will call the MP package manager.

The problem formulated in the previous section in various UNIX / Linux distributions is solved by installing, uninstalling and storing information about installed software on a special program - the package manager. On Debian / Ubuntu, dpkg does this. In fact, this is only one program from a whole set of programs that are relevant to this. dpkg, this is the backend, there are several frontends, apt-get, aptitude, dselect, synaptic.

What does MP do? At least in Debian / Ubuntu, he does the following:

1. Install packages.
2. Removing packages.
3. Dependency tracking (when installing a package, install other packages necessary for this package along with the package)
4. Install packages over the network from remote repositories.
5. Maintaining a local database of installed packages.
6. Maintaining a local database of packages available for installation, including over the network.
7. Updating databases when changing available packages and / or their versions.
8. Correct installation of package updates.
9. Upgrade the whole system.

A lot, how do you think? And the Debian / Ubuntu MP does a really good job of this.

The package itself is usually an archive of files, a package description with several fields, and scripts that are executed when installing and removing the package. On Debian / Ubuntu, the package file has a .deb extension.

6.0 Problems Associated with Package Managers

As you know, nothing happens without problems. The use of MP is also no exception and brings its own problems. Of course, the MP solves much more problems than it brings, but still they need to be known and deal with them. So the problems are:

1. Dependencies. This is a plus of using MP on the one hand. Dependency tracking, on the other hand, is a big job for package maintainers. Today, Debian has 25,113 packages. Some packages require the mandatory installation of others, or vice versa, there are incompatible packages. The result is a large and complex system of dependencies. All this stuff needs to be checked.
2. Incompatible package combinations are sometimes found.
3. Sometimes there are all sorts of cunning ring dependencies.
4. Due to the large number of packages that need to be combined with each other, sometimes the package versions included in the distribution become obsolete.
5. Often in the distribution software is not the version that we need.
6. Despite such a large number of packages, sometimes we need software that is not in the distribution at all.
7. We cannot install our software in the directory where the MP runs, otherwise our software may come into conflict with the software installed by the MP. Simply put, we can overwrite files installed by MP, or vice versa.

7.0 Solving problems related to package managers, hierarchy / usr / local

Recently, almost all the necessary software is available in the distribution package. But still, sooner or later, as a rule, each user / administrator is faced with the need to install software that is not included in the distribution. Usually there are such options:

Problem 1.	It is necessary to recompile the package with other settings.
Problem 2	The necessary software is not in the distribution kit.
Problem 3.	The distribution is not the version we want.

How to be How to do this with the least risk of disturbing the system? There are several options available:

Problem 1.	Solution 1:	Recompile the package with other settings. (This is if you need a package with other settings). There is practically no danger, although dependencies may change compared to the package compiled in the distribution. There is already, clear pepper, you are responsible for the addiction.
Problems 2,3	Solution 2	Build your own package and install in the system.
Problems 1,2,3	Decision 3	Install software collected from source in the hierarchy / usr / local

Since Solution 1 seems to be clear, let's take a closer look at Solution 2 and Solution 3 .

Solution 2: Build your own package and install in the system.

Pros:

1. You can use all the features of the MP listed in section 6.0.
2. If the package is designed to be installed on several machines, you can use the network features of the MP, create your own repository with your packages, and install / update these packages in the standard way for the MP.

Minuses:

1. Using all or only part of the MP capabilities requires additional efforts, sometimes significant. If you do not use them, then what's the point in using MP?
2. If you do not check your package for compatibility, it is very likely that your package will conflict with other packages, and conflicts may be most unexpected.
3. With the mass implementation of this approach (creating your own packages), it becomes possible for a mass of poorly tested and poorly compatible packages to appear on the Internet, as was the case with RPM at one time, I don’t know how it is with them now.

Debian Policy ² (Ubuntu as a derivative of Debian is guided by it too) directly says two things:

• Locally installed software is installed in the / usr / local hierarchy .
• The software installed locally should be safe from overwriting when updating the system (meaning updating via MP).

Thus, we can conclude that for software that is not in the repository, which is installed locally on this machine, or on a small number of machines, the standard way is to install in / usr / local .

If the software is intended for distribution and installation on Debian / Ubuntu systems, the best way is to package it in .deb packages. BUT! In this case, you are responsible for the dependencies, conflicts, and updates of this package.

8.0 Problems with the / usr / local hierarchy

1. As you know, any solution brings new problems. Not an exception and the solution outlined in the previous section. Everything is fine, as long as we installed one package in / usr / local , as soon as there are more than one package, a poorly managed bunch of files appears, with problems. described in section 4.0.
2. Another problem is the problem with the / usr / local / var directory . As you know, the program data files that can change during program operation (logs, database PIDs, and so on) are written to the / var directory . Often
   / var is done on a separate partition, but now it turned out that the mutable files were in / usr / local / var .

9.0 Solving problems related to the / usr / local hierarchy, possible workarounds

The solution to the problem 1. is described in my article ⁵ , therefore I will not describe the solution in detail. In short, each program is placed in its own hierarchy in the / usr / local / stow directory , and then, a special manager (xstow), places symbolic links to the program files.

Solution of the problem 2. This problem will have to be solved by hands. If you used the stow program, you can, for example, make a directory with a unique name in the / var hierarchy , for example / var / local_var , and then make links to it. But this is only worth doing if there really is such a need.

10.0 Hierarchy / opt

I install third-party software into the / opt hierarchy for Policy ² . They install their software in a directory of the form / opt / package or / opt / provider .

11.0 User Issues, / home hierarchy

Again, according to Policy ² , user files are stored in the / home hierarchy , in directories of the form / home / username . If we talk about installing software, then it can be installed here. moreover, how to do this is left to the discretion of the user.

When to do it?

• The user does not have administrator rights on the computer.
• Temporarily, "to see."
• Own scripts.

I usually use a mix of directories and packages by file layout. For software that works constantly, as a rule, these are my scripts; I create the directories / home / username / bin / home / username / etc / home / username / var . The software that I just want to watch or play just compiles in the directory / home / username / sw / softname and just starts from there.

12.0 Other uses of the / home hierarchy

The / home hierarchy is often used to host files related to large tasks performed on a computer. For example, a project is being developed with the name projectname . Then they start up a user with the name projectname in the system and everything related to it is placed in the / home / projectname directory

13.0 Other OS

Once Vovochkin’s mother said to her dad:
“Don't you think that it’s time for our Vovochka
to talk about sex?”
Dad thought, and answered:
"you are probably right, only somehow uncomfortable."
Mom: "Well, tell me on the example of butterflies."
Dad called Vovochka and says:
"Vovochka, do you remember we were in a
brothel?"
“I remember.”
"Well, here: the butterflies are exactly the same."


Joke

Freebsd

On FreeBSD, in principle, everything is the same, with slight differences. The files of the base system are located there like the Linux family of Debian / Ubuntu in the main hierarchy. The binary packages there correspond to two concepts, packages and ports.

Packages are about the same as packages in Debian / Ubuntu. As a package manager, a set of commands is used there to work with packages (and ports).

A port is a set of auxiliary files that allows you to compile and install sources in the system. It allows you to automatically download source codes from media (for example, CDROM) or download from the Internet, compile and install in the system.

Both packages and ports are installed in the / usr / local hierarchy .

In the same way as in Debian / Ubuntu, a situation may arise when there is neither a package nor a port for software. Then, of course, you need to install software from source. To do this, you can either make a port for this software, or simply compile and install the sources in the / usr / local hierarchy . Of course, since there is a lot of software installed in FreeBSD, you need to be careful, because the likelihood of conflict because of this increases.

Well, returning to the methods described in article ⁵ , I think that if you do not create your own port, you can use the xstow program to maintain order. Although BSD-ists may have their favorite methods, unknown to me, let knowledgeable comrades correct me if I am mistaken.

Windows family

In the Windows family of operating systems, as a rule, an approach is used to host software when each software package has its own hierarchy. There are some shifts towards the layout of files by destination, but this is hindered by:

- Lack of a single tree by the file system (letters indicating partitions, C :, D: etc.).
- The tradition.
- Requirements for compatibility with proven software.

Conclusion

I hope the article will be useful. In principle, it would be possible in another article to analyze in more detail the purpose of other UNIX / Linux FS directories and hierarchies.

List of abbreviations used

.deb	Debian / Ubuntu Software Package File Extension
dpkg	dpkg is the software that is the core of the Debian package management system.
PID	Process id
RPM	Red Hat Package Manager - Red Hat Package Manager
Db	Database
MP	Package manager
OS	operating system
BY	Software
repositories	Software storage with access using MP
FS	File system

Literature

1. Linux Standard File System Hierarchy
2. Filesystem Hierarchy Standard (Debian Policy Addendum)
3. http://en.wikipedia.org/wiki/RPM 4.http
: //en.wikipedia.org / wiki / Dpkg
5. Linux: Installing non-distribution programs using the xstow
6 manager . man hier