March 7, 2013 at 15:51Optimization of assembly of a large project
Almost all developers face the problem of increasing project assembly time at least once a year. Build is not a fast task, which is especially unpleasant when using the Continuous Integration practice with its constant rebuilding and related activities. Long assembly negates all the advantages of continuous integration, and a simple increase in computing power does not always give the desired effect.
In the process of developing and further developing a large heterogeneous project, we also encountered optimization problems. But there must be ways to reduce assembly time? We decided to find a way to solve this problem. Collect in 60 seconds! .. About what we eventually managed to achieve, and we will discuss it in our topic.
The key to successful project optimization is its automation. We used the well-known CMake automation system, on which, as on the foundation of the house, we built various technologies.
Here is what we had at the beginning of the journey:
All this was going to 12 minutes.
After optimizing the code based on the results of static analysis, removing unnecessary #include directives, rebuilding executable modules, dynamic and static libraries, the assembly time was reduced to 8 minutes. Further, we will focus on optimizing the build process when all possible code improvements are already implemented.
To achieve maximum speed, we used two approaches to optimization: hardware and software. As we will show later, the former does not exclude the latter at all, but supplements it.
A lot has been written about typical VisualStudio customization methods on the Internet, so let's touch them in passing.
More interesting is the technology known as UnityBuild . This is one of the most effective software optimization methods, accelerating assembly by 30%. In the process, we slightly improved this technology and named a new, improved version of Smart UnityBuild.
The standard technology works as follows: each unity * .cmake file of each project uses the unity_build.cmake module, which is directed to a folder containing all * .cpp files, processes them and merges them into a new “common” * .cpp file. Next, a * .vcproj file is created containing this single * .cpp.
Our version does not work quite like that. In some cases (in particular, when working with a large project), creating one common * .cpp file is not the best option. Therefore, we decided to generate several files instead of one large one.
Schematically, it looks like this:
As a result, the use of our Smart UnityBuild technology allowed us to speed up the assembly up to 4 minutes. 40 sec
Obviously, using PCH in conjunction with UnityBuild technology, which is called head-on, will not do anything. But still there is a way to fool VisualStudio. To do this, you need to artificially transfer the files used in the project several times (we have * .boost and * .spl) into a separate project, assemble the Precompiled Headers from it and put them in the main project, as a result of VisualStudio it will “think” that these are real PCH. Using Unity Build technology with Precompiled Headers and an LP key, you can achieve a significant reduction in build time: in our case, from 12 minutes. up to 4 minutes 38 sec (without LP key - 5 min. 53 sec.).
It is possible to reduce the assembly time of a project, of course, not only combining various technologies. You can also significantly speed up the process by optimizing the server hardware.
The best result in terms of build time was achieved using 7 SSD disks in the Raid0-array and RAMDisk: the duration of the project was about 6 minutes.
The increase in speed is more than twice compared to the standard value of time (7 min. 45 sec.) Gives the use of 32 processors with Hyper Threading technology (3 min. 22 sec.).
By combining all the tools from Visual Studio options to using SSD drives, we were able to reduce the build time of the project from 12 minutes to 1 minute. 45 sec Thus, the stated goal - to meet 60 seconds - remained unfulfilled, but we were not upset :) Reducing the time spent on assembling the project by more than 6 times is a good result.
Here's what the build progress of our project looks like:
The diagram shows that several time-consuming processes interfere with achieving even greater speed. This is a link for large modules, which is impossible to parallelize (only partially there is an attempt to do this in MSVS 2012), but you can break them into several small ones programmatically, reduce the number and size of static libraries.
That, in fact, is all. Our goal was not to embrace the immensity, and some technical aspects of optimization were not included in the article. We are ready to disclose details and answer all questions (including the source code for cmake) in the comments. Thank you for your attention!
PS The thoughts expressed in this topic formed the basis of the presentation that Viktor Strelkov (Head of Research and Quality Control Department of Positive Technologies) presented at the CEE-SECR conference in November 2012. The recording of the broadcast of this performance can be viewed on the event website .
In the process of developing and further developing a large heterogeneous project, we also encountered optimization problems. But there must be ways to reduce assembly time? We decided to find a way to solve this problem. Collect in 60 seconds! .. About what we eventually managed to achieve, and we will discuss it in our topic.
Start
The key to successful project optimization is its automation. We used the well-known CMake automation system, on which, as on the foundation of the house, we built various technologies.
Here is what we had at the beginning of the journey:
- 176 projects C ++ and C # (in the ratio of approximately 60% to 40%);
- 100 MB source
- 18,000 files (> 3 million lines);
- MSVS;
- server (8 CPU, regular disk on 10 RAID raid, 12 GB of RAM);
- virtual machine.
All this was going to 12 minutes.
After optimizing the code based on the results of static analysis, removing unnecessary #include directives, rebuilding executable modules, dynamic and static libraries, the assembly time was reduced to 8 minutes. Further, we will focus on optimizing the build process when all possible code improvements are already implemented.
To achieve maximum speed, we used two approaches to optimization: hardware and software. As we will show later, the former does not exclude the latter at all, but supplements it.
Software method
Typical methods
A lot has been written about typical VisualStudio customization methods on the Internet, so let's touch them in passing.
- The / MP switch, which allows you to run multiple compilation processes at once.
- The / Z7 switch, which disables the generation of debugging information, resulting in faster compilation.
More interesting is the technology known as UnityBuild . This is one of the most effective software optimization methods, accelerating assembly by 30%. In the process, we slightly improved this technology and named a new, improved version of Smart UnityBuild.
The standard technology works as follows: each unity * .cmake file of each project uses the unity_build.cmake module, which is directed to a folder containing all * .cpp files, processes them and merges them into a new “common” * .cpp file. Next, a * .vcproj file is created containing this single * .cpp.
Our version does not work quite like that. In some cases (in particular, when working with a large project), creating one common * .cpp file is not the best option. Therefore, we decided to generate several files instead of one large one.
Schematically, it looks like this:
As a result, the use of our Smart UnityBuild technology allowed us to speed up the assembly up to 4 minutes. 40 sec
UnityBuild + PCH (Incredibuild)
Obviously, using PCH in conjunction with UnityBuild technology, which is called head-on, will not do anything. But still there is a way to fool VisualStudio. To do this, you need to artificially transfer the files used in the project several times (we have * .boost and * .spl) into a separate project, assemble the Precompiled Headers from it and put them in the main project, as a result of VisualStudio it will “think” that these are real PCH. Using Unity Build technology with Precompiled Headers and an LP key, you can achieve a significant reduction in build time: in our case, from 12 minutes. up to 4 minutes 38 sec (without LP key - 5 min. 53 sec.).
Hardware method
It is possible to reduce the assembly time of a project, of course, not only combining various technologies. You can also significantly speed up the process by optimizing the server hardware.
SSD RAID0 x 7
The best result in terms of build time was achieved using 7 SSD disks in the Raid0-array and RAMDisk: the duration of the project was about 6 minutes.
32 CPU x 3 GHz
The increase in speed is more than twice compared to the standard value of time (7 min. 45 sec.) Gives the use of 32 processors with Hyper Threading technology (3 min. 22 sec.).
Total
By combining all the tools from Visual Studio options to using SSD drives, we were able to reduce the build time of the project from 12 minutes to 1 minute. 45 sec Thus, the stated goal - to meet 60 seconds - remained unfulfilled, but we were not upset :) Reducing the time spent on assembling the project by more than 6 times is a good result.
Here's what the build progress of our project looks like:
The diagram shows that several time-consuming processes interfere with achieving even greater speed. This is a link for large modules, which is impossible to parallelize (only partially there is an attempt to do this in MSVS 2012), but you can break them into several small ones programmatically, reduce the number and size of static libraries.
That, in fact, is all. Our goal was not to embrace the immensity, and some technical aspects of optimization were not included in the article. We are ready to disclose details and answer all questions (including the source code for cmake) in the comments. Thank you for your attention!
PS The thoughts expressed in this topic formed the basis of the presentation that Viktor Strelkov (Head of Research and Quality Control Department of Positive Technologies) presented at the CEE-SECR conference in November 2012. The recording of the broadcast of this performance can be viewed on the event website .