TL; DR . In this article, we explore hardening schemes that work out of the box on five popular Linux distributions. For each, we took the default kernel configuration, downloaded all the packages, and analyzed the protection schemes in the attached binary files. We consider the distributions of OpenSUSE 12.4, Debian 9, CentOS, RHEL 6.10 and 7, as well as Ubuntu 14.04, 12.04 and 18.04 LTS.

The results confirm that even basic schemes, such as stack canaries and position-independent code, are not yet used by everyone. The situation is even worse for compilers when it comes to protecting against vulnerabilities like stack clash, which got into the spotlight in January after publishing vulnerability information in systemd. But not everything is so hopeless. In a significant part of the binaries, basic protection methods are implemented, and their number is growing from version to version.

Verification showed that the largest number of protection methods is implemented in Ubuntu 18.04 at the OS and application level, followed by Debian 9. On the other hand, in OpenSUSE 12.4, CentOS 7 and RHEL 7, basic protection schemes are also implemented, and stack collision protection is applied even more widely. with a much denser set of packages by default.

Introduction

It is difficult to provide high quality software. Despite the huge number of advanced tools for static code analysis and dynamic analysis at runtime, as well as significant progress in the development of compilers and programming languages, modern software still suffers from vulnerabilities that are constantly exploited by cybercriminals. The situation is even worse in ecosystems that include legacy code. In such cases, we not only face the eternal problem of searching for possible exploited errors, but are also limited by rigid backward compatibility frameworks, which often require maintaining limited, and even worse, vulnerable or buggy code.

This is where hardening methods come into play. We cannot prevent some types of errors, but we can make the life of an attacker more difficult and partially solve the problem by preventing or preventing the operation of these errors. Such protection is used in all modern operating systems, however, the methods vary greatly in complexity, efficiency, and performance: from stack canaries and ASLR to full-fledged CFI and ROP protections . In this article, we will consider what protection methods are used in the most popular Linux distributions in the default configuration, and also study the properties of binaries that are distributed through the package management systems of each distribution.

CVE and security

We've all seen articles with titles like “Most Vulnerable Applications of the Year” or “Most Vulnerable Operating Systems”. Typically, they provide statistics on the total number of vulnerability records such as CVE (Common Vulnerability and Exposures) obtained from the National Vulnerability Database (NVD) from NIST and other sources. Subsequently, these applications or operating systems are ranked by the number of CVEs. Unfortunately, although CVEs are very useful for tracking issues and informing vendors and users, they say little about the real security of software.

As an example, consider the total number of CVEs over the past four years for the Linux kernel and the five most popular server distributions, namely Ubuntu, Debian, Red Hat Enterprise Linux, and OpenSUSE.


Fig. 1

What does this chart tell us? Does more CVE mean that one distribution is more vulnerable than another? No answer. For example, in this article you will see that Debian implements more stringent security mechanisms compared to, say, OpenSUSE or RedHat Linux, and yet Debian has more CVE. However, they do not necessarily mean weakened security: even having a CVE does not say whether the vulnerability is exploitable . The severity scores give an idea of ​​how likely the exploitation of the vulnerability is, but ultimately the exploitability is largely dependent on the protection present in the affected systems, as well as on the resources and capabilities of the attackers. Moreover, the lack of CVE reports says nothing about othersunregistered or unknown vulnerabilities. The difference in CVE can be explained not by the quality of the software, but by other factors, including the resources allocated for testing, or the size of the user base. In our example, more Debian CVEs may simply indicate that Debian is delivering more software packages.

Of course, the CVE system provides useful information that allows you to create appropriate protections. The better we understand the reasons for the failure of the program, the easier it is to identify possible methods of operation and to develop appropriate detection and response mechanisms . In fig. Figure 2 shows the categories of vulnerabilities for all distributions over the past four years ( source) It’s immediately obvious that most CVEs fall into the following categories: denial of service (DoS), code execution, overflow, memory corruption, information leakage (exfiltration), and privilege escalation. Although many CVEs are covered several times in different categories, in general, the same problems persist from year to year. In the next part of the article, we will evaluate the use of various protection schemes to prevent the exploitation of these vulnerabilities.


Fig. 2

Tasks

In this article, we intend to answer the following questions:

• What is the security of various Linux distributions? What defense mechanisms exist in the kernel and user space applications?
  
• How has the adoption of protection mechanisms for various distributions changed over time?
  
• What are the average dependencies of packages and libraries for each distribution?
  
• What protections are implemented for each binary?

Choosing Distributions

It turns out that it is difficult to find accurate statistics on distributions installations, since in most cases the number of downloads does not indicate the number of real installations. However, Unix variants make up the majority of server systems (69.2% on web servers, according to statistics from W3techs and other sources), and their share is constantly growing. Thus, for our study, we focused on distributions available out of the box on the Google Cloud platform . In particular, we chose the following OS:

Distribution / version	Core	Build
OpenSUSE 12.4	4.12.14-95.3-default	# 1 SMP Wed Dec 5 06:00:48 UTC 2018 (63a8d29)
Debian 9 (stretch)	4.9.0-8-amd64	# 1 SMP Debian 4.9.130-2 (2018-10-27)
CentOS 6.10	2.6.32-754.10.1.el6.x86_64	# 1 SMP Tue Jan 15 17:07:28 UTC 2019
CentOS 7	3.10.0-957.5.1.el7.x86_64	# 1 SMP Fri Feb 1 14:54:57 UTC 2019
Red Hat Enterprise Linux Server 6.10 (Santiago)	2.6.32-754.9.1.el6.x86_64	# 1 SMP Wed Nov 21 15:08:21 EST 2018
Red Hat Enterprise Linux Server 7.6 (Maipo)	3.10.0-957.1.3.el7.x86_64	# 1 SMP Thu Nov 15 17:36:42 UTC 2018
Ubuntu 14.04 (Trusty Tahr)	4.4.0–140-generic	# 166 ~ 04/14/1-Ubuntu SMP Sat Nov 17 01:52:43 UTC 20 ...
Ubuntu 16.04 (Xenial Xerus)	4.15.0-1026-gcp	# 27 ~ 16.04.1-Ubuntu SMP Fri Dec 7 09:59:47 UTC 2018
Ubuntu 18.04 (Bionic Beaver)	4.15.0-1026-gcp	# 27-Ubuntu SMP Thu Dec 6 18:27:01 UTC 2018

Table 1

Analysis

We will examine the default kernel configuration, as well as the properties of packages available through the package manager of each distribution package out of the box. Thus, we only consider packages from the default mirrors of each distribution, ignoring packages from unstable repositories (for example, 'testing' mirrors in Debian) and third-party packages (for example, Nvidia packages from standard mirrors). In addition, we do not consider custom kernel compilations or advanced security configurations.

Kernel configuration analysis

We used an analysis script based on the free kconfig checker . We consider the protection options out of the box for the named distributions and compare them with the list from the Kernel Self-Defense Project (KSPP). For each configuration parameter, Table 2 describes the desired setting: a check mark is for distributions that comply with the KSSP recommendations (for an explanation of the terms see here ; in future articles we will describe how many of these protection methods appeared and how to hack the system in their absence). In general, the newer kernels have more stringent settings out of the box. For example, CentOS 6.10 and RHEL 6.10 on the 2.6.32 kernel do not have most of the critical functions implemented in new kernels, such as SMAP





, strong RWX permissions, address randomization, or copy2usr protection. It should be noted that many of the configuration options from the table are absent in older versions of the kernel and are not applicable in reality - this is still indicated in the table as the lack of proper protection. Similarly, if the configuration parameter is not available in this version, and for security, this parameter must be disabled, this is considered a reasonable configuration.

Another point when interpreting the results: some kernel configurations that increase the attack surface can be used simultaneously for security. Such examples include uprobes and kprobes, kernel modules, and BPF / eBPF. Our recommendation is to use the above mechanisms to provide real protection, since they are not trivial to use, and their operation assumes that malicious actors are already entrenched in the system. But if these options are enabled, the system administrator must actively monitor the abuse.

Studying further the entries in Table 2, we see that modern kernels provide several options for protecting against exploitation of such vulnerabilities as information leakage and stack / heap overflow. However, we notice that even the most recent popular distributions have not yet implemented more sophisticated protection (for example, with grsecurity patches ) or modern protection against code reuse attacks (for example, combining randomization with schemes like R ^ X for code ). Even worse, even these more advanced defenses do not protect against a full range of attacks. Thus, it is crucial for system administrators to complement smart configurations with solutions that offer detection and prevention of exploits at runtime.

Application analysis

It is not surprising that different distributions have different characteristics of packages, compilation options, library dependencies, etc. Differences exist even for related distributions and packages with a small number of dependencies (for example, coreutils in Ubuntu or Debian). To evaluate the differences, we downloaded all available packages, extracted their contents and analyzed the binary files and dependencies. For each package, we tracked the other packages on which it depends, and for each binary we tracked its dependencies. This section summarizes the findings.

Distributions

In total, we downloaded 361,556 packages for all distributions, extracting only packages from the default mirrors. We ignored packages without ELF executable files, such as source codes, fonts, etc. After filtering, 129 569 packages were left containing a total of 584 457 binary files. Distribution of packages and files among distributions is shown in fig. 3.


Fig. 3

You may notice that the more modern the distribution, the more packages and binary files it contains, which is logical. At the same time, Ubuntu and Debian packages include much more binary files (both executable and dynamic modules and libraries) than CentOS, SUSE and RHEL, which potentially affects the attack surface of Ubuntu and Debian (it should be noted that the numbers reflect all binaries of all versions package, that is, some files are analyzed several times). This is especially important when considering the dependencies between packages. Thus, a vulnerability in the binary of a single package can affect many parts of the ecosystem, just as a vulnerable library can affect all binary files importing it. As a reference point, let's look at the distribution of the number of dependencies among packages in various OSs: Fig. 4




In almost all distributions, 60% of packages have at least 10 dependencies. In addition, some packages have more dependencies (over 100). The same applies to inverse package dependencies: as expected, several packages are used by many other packages in the distribution, so vulnerabilities in these few favorites have a high risk. As an example, the following table lists 20 packages with the maximum number of inverse dependencies in SLES, Centos 7, Debian 9, and Ubuntu 18.04 (each box indicates the package and the number of inverse dependencies).


Table 3

Interesting fact. Although all analyzed OSs are built for the x86_64 architecture, while most packages have the architecture defined as x86_64 and x86, packages often contain binaries for other architectures, as shown in Fig. 5.


Fig. 5

In the next section, we will delve into the characteristics of the analyzed binaries.

Binary Protection Statistics

As an absolute minimum, you need to study the basic set of protection options for existing binary files. Several Linux distributions come with scripts that perform such checks. For example, in Debian / Ubuntu there is such a script. Here is an example of his work:

$ hardening-check $(which docker)
/usr/bin/docker:
 Position Independent Executable: yes
 Stack protected: yes
 Fortify Source functions: no, only unprotected functions found!
 Read-only relocations: yes
 Immediate binding: yes

The script checks five protection functions :

• Position Independent Executable (PIE): indicates whether the program text section can be moved in memory to achieve randomization if ASLR is enabled in the kernel.
  
• Stack Protected: whether stack canaries are included to protect against stack collision attacks.
  
• Fortify Source: whether unsafe functions (e.g. strcpy) are replaced by their safer counterparts, and calls checked in runtime are replaced by their non-verified counterparts (e.g. memcpy instead of __memcpy_chk).
  
• Read-only relocations (RELRO): Whether the entries in the movement table are marked as read-only if they worked before the execution started.
  
• Immediate binding: whether the runtime linker allows all movements before starting the program (this is equivalent to full RELRO).

Are the above mechanisms enough? Unfortunately not. There are known ways to circumvent all of the above defenses, but the more stringent the defense, the higher the bar for the attacker. For example, RELRO workarounds are harder to apply if PIE and immediate binding are in effect. Likewise, a full ASLR requires additional work to create a working exploit. However, sophisticated attackers are ready to meet such defenses: their absence will essentially accelerate hacking. Therefore, it is imperative that these measures be considered the necessary minimum .

We wanted to study how many binary files in the distributions in question are protected by these, as well as three more methods:

• An unexecutable bit ( NX ) prevents execution in any region that should not be executable, such as a stack heap, etc.
• RPATH / RUNPATH indicates the execution path used by the dynamic loader to find the appropriate libraries. The first is mandatory for any modern system: its absence allows attackers to arbitrarily write the payload into memory and execute it as is. For the second, incorrect configurations of the execution path help in introducing untrusted code, which can lead to a number of problems (for example, escalation of privileges , as well as other problems ).
• Stack collision protection provides protection against attacks that cause the stack to overlap with other areas of memory (such as a heap). Given recent exploits abusing heap collision vulnerabilities in systemd , we found it appropriate to include this mechanism in our data set.

So, without further ado, let's move on to the numbers. Tables 4 and 5 contain a squeeze of analysis of executable files and libraries of various distributions, respectively.

• As you can see, NX protection is implemented everywhere, with rare exceptions. In particular, its lower use in Ubuntu and Debian distributions can be noted in comparison with CentOS, RHEL and OpenSUSE.
  
• Stack canaries are not available in many places, especially in distributions with old kernels. Some progress has been seen in recent distributions of Centos, RHEL, Debian, and Ubuntu.
  
• With the exception of Debian and Ubuntu 18.04, most distributions have poor PIE support.
  
• Stack collision protection is poorly implemented in OpenSUSE, Centos 7 and RHEL 7 and is practically absent from the rest.
  
• All distributions with modern kernels have some RELRO support, with Ubuntu 18.04 leading the way, with Debian taking the second place.

As already mentioned, the metrics in this table are average over all versions of the binary file. If you look only at the latest versions of files, then the numbers will be different (for example, see the progress of Debian with the implementation of PIE ). Moreover, most distributions usually when calculating statistics check the protection of only a few functions in binary code, and in our analysis the true percentage of strengthened functions is indicated. Therefore, if 5 out of 50 functions are protected in the binary, we will give it a rating of 0.1, which corresponds to 10% of the strengthened functions.


Table 4. Protection characteristics for executable files shown in fig. 3 (implementation of the corresponding functions as a percentage of the total number of executable files)


Table 5. Protection characteristics for the libraries shown in Fig. 3 (implementation of the corresponding functions as a percentage of the total number of libraries)

So is there progress? There is definitely: it can be seen from the statistics for individual distributions (for example, Debian ), as well as from the tables above. As an example in fig. Figure 6 shows the implementation of defense mechanisms in three consecutive Ubuntu LTS 5 distributions (we have omitted the stack collision protection statistics). We notice that from version to version more and more files support stacked canaries, and also sequentially more and more binary files come with full RELRO protection. Fig. 6




Unfortunately, a number of executable files in different distributions still do not have any of the above protections. For example, looking at Ubuntu 18.04, you can see the ngetty binary (getty replacement), as well as the mksh and lksh shells, the picolisp interpreter, the nvidia-cuda-toolkit packages (a popular package for GPU-accelerated applications such as machine learning frameworks) and klibc -utils. Similarly, the mandos-client binary (an administrative tool that allows you to automatically restart machines with encrypted file systems), as well as rsh-redone-client (re-implementing rsh and rlogin) are delivered without NX protection, although they have SUID rights :(. In addition, Several suid binaries do not have basic protection, such as stack canaries (for example, the Xorg.wrap binary from the Xorg package).

Summary and concluding remarks

In this article, we highlighted several security features of modern Linux distributions. The analysis showed that the latest Ubuntu LTS distribution (18.04) has on average the strongest OS and application level protection among distributions with relatively new kernels, such as Ubuntu 14.04, 12.04, and Debian 9. However, the CentOS, RHEL, and OpenSUSE distributions discussed in our kit by default, a denser set of packages is issued, and the latest versions (CentOS and RHEL) have a higher percentage of stack collision protection compared to competitors based on Debian (Debian and Ubuntu). Comparing the CentOS and RedHat versions, we notice big improvements in the implementation of stack canaries and RELRO from versions 6 to 7, but on average CentOS has more features than RHEL. Generally,

Finally, it should be noted: although we did the research manually, there are many security tools (for example, Lynis , Tiger , Hubble ) that perform analysis and help to avoid unsafe configurations. Unfortunately, even strong protection in reasonable configurations does not guarantee the absence of exploits. That is why we are firmly convinced that it is vital to ensure reliable monitoring and prevention of attacks in real time , focusing on operational models and preventing them.