March 1, 2009 at 17:21File Compression in the Multi-Core Era
- Transfer
We did a daily big backup for Stack Overflow , so again I played around a bit with file compression.
Our server has the latest 64-bit version of 7zip (4.64) installed . I believe that a dual-core processor is enough for a desktop system. But with servers everything is simple: the more cores, the more fun! The server has two quad-core processors - only eight cores, so I was a little disappointed to find that neither RAR nor 7zip use more than two cores.
But, even using only two cores for compression, the 7zip algorithm is incredibly efficient, and recently it has become quite fast. I used to recommend using RAR instead of Zip. Given the free, unlike RAR, and the effectiveness of 7zip, now it will be logical to choose it.
Here are a couple of my tests for compressing a 4.76 GB database backup file (I used a server with two quad-core 2.5 GHz Xeon E5420 processors on board):
If you think that since 7zip shows good results with compression rates of maximum and ultra, then with ultra-plus the result will be completely unbelievable , I have to disappoint you. There are certain reasons why archiver manufacturers set the default compression ratio to normal. If the compression ratio is reduced, the size of the output archive increases sharply, while an increase in the compression ratio leads to a relatively small decrease in the size of the output archive in exchange for a large increase in the compression time.
Now let's see what happens if I switch 7zip to using bzip2 :
We will compress the same file (4.76 GB) on the same machine:
Why is bzip2 so much faster than 7zip? It's simple:
loading the processor when using 7zip
loading the processor when using bzip2
Bzip2 uses more than two cores for parallelization. I don’t know how many kernels he can use, but in the drop-down menu 7zip offers no more than sixteen kernels for bzip2. Our server has eight cores, so I chose so many during the test.
Unfortunately, increasing the speed of bzip2 is pointless with a high compression ratio - the difference between normal, maximum, and ultra is a miserable 0.06%. The algorithm perfectly scales in time, but practically does not scale in size of the output file. This is very strange because it is precisely to reduce the size that I would like to spend the time won by parallelization.
But even a minimal reduction in size can make sense, depending on the circumstances:
For example, if you compress a file to send it over the network, n = 1, so the compression time significantly affects the total time. If you want to upload a file on the Internet, n is large, so a large compression time will have little effect on the total time.
After all, slow networks work well with slow but efficient algorithms, while fast networks need fast but possibly less efficient algorithms.
On the other hand, the ability to compress a five-gigabyte file five times in two minutes is impressive. Therefore, the thought of how quickly the 7zip algorithm would work if I rewrote it and parallelized it to work on more than two cores does not leave me.
Our server has the latest 64-bit version of 7zip (4.64) installed . I believe that a dual-core processor is enough for a desktop system. But with servers everything is simple: the more cores, the more fun! The server has two quad-core processors - only eight cores, so I was a little disappointed to find that neither RAR nor 7zip use more than two cores.
But, even using only two cores for compression, the 7zip algorithm is incredibly efficient, and recently it has become quite fast. I used to recommend using RAR instead of Zip. Given the free, unlike RAR, and the effectiveness of 7zip, now it will be logical to choose it.
Here are a couple of my tests for compressing a 4.76 GB database backup file (I used a server with two quad-core 2.5 GHz Xeon E5420 processors on board):
| 7zip | fastest | 5 min | 14 MB / sec | 973 MB |
| 7zip | fast | 7 min | 11 MB / sec | 926 MB |
| 7zip | normal | 34 min | 2.5 MB / sec | 752 MB |
| 7zip | maximum | 41 min | 2.0 MB / sec | 714 MB |
| 7zip | ultra | 48 min | 1.7 MB / sec | 698 MB |
Now let's see what happens if I switch 7zip to using bzip2 :
We will compress the same file (4.76 GB) on the same machine:
| bzip2 | fastest | 2 min | 36 MB / sec | 1092 MB |
| bzip2 | fast | 2.5 min | 29 MB / sec | 1011 MB |
| bzip2 | normal | 3.5 min | 22 MB / sec | 989 MB |
| bzip2 | maximum | 7 min | 12 MB / sec | 987 MB |
| bzip2 | ultra | 21 min | 4 MB / sec | 986 MB |
loading the processor when using 7zip
loading the processor when using bzip2
Bzip2 uses more than two cores for parallelization. I don’t know how many kernels he can use, but in the drop-down menu 7zip offers no more than sixteen kernels for bzip2. Our server has eight cores, so I chose so many during the test.
Unfortunately, increasing the speed of bzip2 is pointless with a high compression ratio - the difference between normal, maximum, and ultra is a miserable 0.06%. The algorithm perfectly scales in time, but practically does not scale in size of the output file. This is very strange because it is precisely to reduce the size that I would like to spend the time won by parallelization.
But even a minimal reduction in size can make sense, depending on the circumstances:
итоговое время = время сжатия + n * (размер архива / скорость сети + время распаковки)For example, if you compress a file to send it over the network, n = 1, so the compression time significantly affects the total time. If you want to upload a file on the Internet, n is large, so a large compression time will have little effect on the total time.
After all, slow networks work well with slow but efficient algorithms, while fast networks need fast but possibly less efficient algorithms.
On the other hand, the ability to compress a five-gigabyte file five times in two minutes is impressive. Therefore, the thought of how quickly the 7zip algorithm would work if I rewrote it and parallelized it to work on more than two cores does not leave me.