The Impact of Specter, Meltdown, and Foreshadow Protection on Linux 4.19 Performance
- Transfer
One of the most frequently asked questions lately is how Meltdown / Specter protection affects Linux performance, and now L1TF / Foreshadow protection. The beginning of the development of the Linux 4.19 kernel this month added fuel to the fire, not only for x86_64, but also for POWER / s390 / ARM. To get a general idea of the effect of patches on performance, I tested three Intel Xeon systems and two AMD EPYC systems, as well as a virtual machine on each side, to evaluate the performance of the Linux 4.19 default with and without corresponding patches.
All machines had a system with a Linux kernel 4.19-rc1 released last weekend. On Intel, the corresponding patches include isolation of page tables (PTI / KPTI) for Meltdown and various Specter patches against speculative execution of commands, including cleaning the __user pointer, using retpoline via IBPB IBRS_FW, patch for Speculative Store Bypass vulnerability using prctl and seccomp, and also PTE inversion and conditional cache reset in virtual machines - this is for L1TF / Foreshadow.
By default, the Linux kernel does not provide “full” protection against vulnerabilities by disabling Intel HT / SMT support, so keep this in mind if you use virtual machines and provide unreliable code or users with access to the VM. If you choose full protection and disable SMT, thenThe performance impact is much more noticeable due to halving the number of threads available. Cloud service providers seem to simply set up schedulers to prevent SMT flows from passing through users. This avoids the obvious huge costs of disabling Hyper Threading. So now we compare only the protection of the stock / default core.
On AMD EPYC, by default, protection is implemented only for the corresponding vulnerabilities that affect them: these are cleaning the __user pointer for Specter V1, AMD Retpoline IBPB for Specter V2, and disabling the Speculative Store Bypass (SSBD) for Specter V4.
After testing all configurations on the Linux 4.19-rc1 stock kernel, the tests were repeated using various security switches in runtime.. All systems were tested with Ubuntu 18.04.1 LTS x86_64 with Linux kernel 4.19-rc1 via Ubuntu Mainline Kernel PPA, latest microcodes / BIOS, GCC 7.3 and EXT4 file system.
System configurations in the test:
Obviously, the configurations of the machines are different and they are not intended to compare with each other, namely, to test the on / off protection against processor vulnerabilities on the Linux 4.1 kernel. Therefore, for clarity, all data are normalized by the performance of each system. All tests from the Phoronix Test Suite .
For the article selected tests that are relevant to Specter / Meltdown, that is, with intensive input-output or kernel interactions. The load is just on the CPU and does not greatly depend on the processor cache.
The CompileBench profile is probably the easiest way to show the Specter / Meltdown influence. On the Linux 4.19 kernel, when activating protection, Intel processors demonstrate a 7–16% decrease in performance, and AMD processors - by 3–4%.
The situation is similar in subtests with reading compiled tree. Intel processors have a 14–15% decrease in performance, while AMD has a 4–5% decrease in performance.
In real tasks like compiling a Linux kernel, the performance difference will be around 2%.
The Hackbench kernel planner benchmark also suffers from the activation of protection. In Intel processors, performance drops by about 20%, except for Xeon Gold. There is not much difference on AMD EPYC systems.
The PostgreSQL database server is one of the real applications that has suffered from performance degradation after installing protection on processors. In this particular test, the difference for Intel processors was 5–8%, and for EPYC - 1%.
Another real performance degradation program due to Specter / Meltdown is the GIMP image editor. The difference for Intel is 5−10%, for AMD - 0−2%.
The Redis DBMS on the Intel Skylake E3 v5 system slows down by 11%, and on other Intel processors by 5-7%, for AMD EPYC systems the difference is 1-5%.
The Nginx web server on the tested Xeon systems on the Linux 4.19 kernel showed a difference in performance up to 20%, and on AMD EPYC - from 1-2% to 6%.
Similarly, the Apache web server with the default installation runs much slower after activating protection on Intel processors and with virtually no changes on AMD processors.
In the OSBench file creation speed test, the system on Intel Xeon slows down by 13–16%, and the EPYC system - by 6−9%.
In the test for creating threads, there is also a noticeable difference between Intel and AMD processors.
When you run programs and create processes, there is a smaller but noticeable difference in the operation of the Linux 4.19 kernel by default compared to disabled protection.
This is the situation with the performance of processors on the Linux 4.19 kernel after installing patches. Keep in mind that if your system (systems) is open to unreliable users / code, especially in virtual machines, then protection may require additional actions, such as l1tf = full, up to disabling SMT / HT or mandatory resetting the L1 cache, which is more reduce system performance. The effect of these protections on L1TF / Foreshadow is described in more detail in the last article .
Perhaps in the future we will carry out similar tests on Linux 4.19 with desktop processors and corresponding workloads.
All machines had a system with a Linux kernel 4.19-rc1 released last weekend. On Intel, the corresponding patches include isolation of page tables (PTI / KPTI) for Meltdown and various Specter patches against speculative execution of commands, including cleaning the __user pointer, using retpoline via IBPB IBRS_FW, patch for Speculative Store Bypass vulnerability using prctl and seccomp, and also PTE inversion and conditional cache reset in virtual machines - this is for L1TF / Foreshadow.
By default, the Linux kernel does not provide “full” protection against vulnerabilities by disabling Intel HT / SMT support, so keep this in mind if you use virtual machines and provide unreliable code or users with access to the VM. If you choose full protection and disable SMT, thenThe performance impact is much more noticeable due to halving the number of threads available. Cloud service providers seem to simply set up schedulers to prevent SMT flows from passing through users. This avoids the obvious huge costs of disabling Hyper Threading. So now we compare only the protection of the stock / default core.
On AMD EPYC, by default, protection is implemented only for the corresponding vulnerabilities that affect them: these are cleaning the __user pointer for Specter V1, AMD Retpoline IBPB for Specter V2, and disabling the Speculative Store Bypass (SSBD) for Specter V4.
After testing all configurations on the Linux 4.19-rc1 stock kernel, the tests were repeated using various security switches in runtime.. All systems were tested with Ubuntu 18.04.1 LTS x86_64 with Linux kernel 4.19-rc1 via Ubuntu Mainline Kernel PPA, latest microcodes / BIOS, GCC 7.3 and EXT4 file system.
System configurations in the test:
- Intel Xeon E3-1280 v5 Skylake on the MSI Z170A SLI PLUS motherboard, 16 GB DDR4 and 256 GB Toshiba RD400 NVMe SSD.
- Intel Xeon E5-2687W v3 Haswell on MSI X299 SLI PLUS motherboard, 32 GB DDR4 and 80 GB Intel 530 SATA 3.0 SSD.
- Two Intel Xeon Gold 6138 rack Tyan 1U with 96 GB of RAM and Samsung 970 EVO NVMe SSD 256 GB.
- KVM virtual machine on the above dual Xeon Gold server. This VM was the only active process on the machine and was configured to access 80% of the CPU cores / threads (64 threads), 48 GB of RAM and a 118 GB virtual disk. During testing, protection against vulnerabilities was disabled on both the host and VM.
- AMD EPYC 7601 on a Tyan 2U server with 128 GB of RAM and 280 GB of Intel Optane 900p NVMe SSD.
- KVM virtual machine on the above AMD EPYC 7601 server. It has access to 80% of CPU cores / threads (52 threads), 48 GB RAM and 120 GB virtual disk.
- AMD EPYC 7551 server on a Gigabyte MZ31-AR0 motherboard with 32 GB RAM and Samsung 960 EVO 256GB NVMe SSD.
Obviously, the configurations of the machines are different and they are not intended to compare with each other, namely, to test the on / off protection against processor vulnerabilities on the Linux 4.1 kernel. Therefore, for clarity, all data are normalized by the performance of each system. All tests from the Phoronix Test Suite .
For the article selected tests that are relevant to Specter / Meltdown, that is, with intensive input-output or kernel interactions. The load is just on the CPU and does not greatly depend on the processor cache.
The CompileBench profile is probably the easiest way to show the Specter / Meltdown influence. On the Linux 4.19 kernel, when activating protection, Intel processors demonstrate a 7–16% decrease in performance, and AMD processors - by 3–4%.
The situation is similar in subtests with reading compiled tree. Intel processors have a 14–15% decrease in performance, while AMD has a 4–5% decrease in performance.
In real tasks like compiling a Linux kernel, the performance difference will be around 2%.
The Hackbench kernel planner benchmark also suffers from the activation of protection. In Intel processors, performance drops by about 20%, except for Xeon Gold. There is not much difference on AMD EPYC systems.
The PostgreSQL database server is one of the real applications that has suffered from performance degradation after installing protection on processors. In this particular test, the difference for Intel processors was 5–8%, and for EPYC - 1%.
Another real performance degradation program due to Specter / Meltdown is the GIMP image editor. The difference for Intel is 5−10%, for AMD - 0−2%.
The Redis DBMS on the Intel Skylake E3 v5 system slows down by 11%, and on other Intel processors by 5-7%, for AMD EPYC systems the difference is 1-5%.
The Nginx web server on the tested Xeon systems on the Linux 4.19 kernel showed a difference in performance up to 20%, and on AMD EPYC - from 1-2% to 6%.
Similarly, the Apache web server with the default installation runs much slower after activating protection on Intel processors and with virtually no changes on AMD processors.
In the OSBench file creation speed test, the system on Intel Xeon slows down by 13–16%, and the EPYC system - by 6−9%.
In the test for creating threads, there is also a noticeable difference between Intel and AMD processors.
When you run programs and create processes, there is a smaller but noticeable difference in the operation of the Linux 4.19 kernel by default compared to disabled protection.
This is the situation with the performance of processors on the Linux 4.19 kernel after installing patches. Keep in mind that if your system (systems) is open to unreliable users / code, especially in virtual machines, then protection may require additional actions, such as l1tf = full, up to disabling SMT / HT or mandatory resetting the L1 cache, which is more reduce system performance. The effect of these protections on L1TF / Foreshadow is described in more detail in the last article .
Perhaps in the future we will carry out similar tests on Linux 4.19 with desktop processors and corresponding workloads.