March 12, 2014 at 09:49Comparison of code analyzers: CppCat, Cppcheck, PVS-Studio, Visual Studio
Authors: Evgeniy Ryzhkov ( EvgeniyRyzhkov ), Andrey Karpov ( Andrey2008 ), Pavel Eremeev ( Paull ), Svyatoslav Razmyslov ( svyatoslavmc ).
We made a detailed comparison of four C / C ++ code analyzers: CppCat, Cppcheck, PVS-Studio and the static analyzer built into Visual Studio 2013. This is a serious study, which we spent about 170 man-hours on, and which, in our opinion , well reflects the picture existing at the moment.
For research, more than ten open-source projects were tested.
The recommended level of detail for diagnostic messages was set:
The comparison methodology is described in detail in the article “How we compared code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ). In it, it will also be possible to find the names of the tested projects, which diagnostic messages were selected, examples of errors found, and so on. It also describes the reason why it was not possible to indicate information about the number of false positives.
The comparison results for the CppCat, Cppcheck, PVS-Studio, and Visual Studio analyzers are presented in the following three tables:
Table 1 - the Number of detected real problems, in pieces
Table 2 - the Number of detected real problems, in percent
Table 3 - the operating time of the analyzers, in minutes
We do not consider the analysis time as an essential parameter of comparison. But they regularly ask questions about the speed of work, so we included time measurements in this article.
As you can see, the fastest is Cppcheck. But keep in mind that this is the total time. On individual projects, other analyzers are faster than Cppcheck. It all depends on the project.
CppCat actually works a little faster than PVS-Studio, since it performs fewer checks. But the difference is so insignificant (tens of seconds) that it makes no sense to mark this in the table.
In total, our team identified 965 unique code fragments in projects that deserve close attention.
The table shows that CppCat and PVS-Studio showed identical results. The fact is that with the recommended settings, the PVS-Studio analyzer displays the same messages as CppCat.
In total, the CppCat analyzer, like PVS-Studio, detects 742 dangerous places in the project code. Cppcheck - 193, Visual Studio - 116.
We can say that the CppCat and PVS-Studio analyzers detect 4 times more errors than Cppcheck and 6 times more than the analyzer from Visual Studio.
The CppCat and PVS-Studio analyzers proved to be clear leaders in the number of defects found and potential defects.
We made a detailed comparison of four C / C ++ code analyzers: CppCat, Cppcheck, PVS-Studio and the static analyzer built into Visual Studio 2013. This is a serious study, which we spent about 170 man-hours on, and which, in our opinion , well reflects the picture existing at the moment.
About Comparison
For research, more than ten open-source projects were tested.
The recommended level of detail for diagnostic messages was set:
- CppCat: all diagnostics (no levels);
- Cppcheck: Errors and Warnings;
- PVS-Studio: Level 1 and Level 2 General Diagnostics;
- Visual Studio: Microsoft Native Recommended Rules.
The comparison methodology is described in detail in the article “How we compared code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ). In it, it will also be possible to find the names of the tested projects, which diagnostic messages were selected, examples of errors found, and so on. It also describes the reason why it was not possible to indicate information about the number of false positives.
Comparison results
The comparison results for the CppCat, Cppcheck, PVS-Studio, and Visual Studio analyzers are presented in the following three tables:
Time
We do not consider the analysis time as an essential parameter of comparison. But they regularly ask questions about the speed of work, so we included time measurements in this article.
As you can see, the fastest is Cppcheck. But keep in mind that this is the total time. On individual projects, other analyzers are faster than Cppcheck. It all depends on the project.
CppCat actually works a little faster than PVS-Studio, since it performs fewer checks. But the difference is so insignificant (tens of seconds) that it makes no sense to mark this in the table.
Diagnostics
In total, our team identified 965 unique code fragments in projects that deserve close attention.
The table shows that CppCat and PVS-Studio showed identical results. The fact is that with the recommended settings, the PVS-Studio analyzer displays the same messages as CppCat.
In total, the CppCat analyzer, like PVS-Studio, detects 742 dangerous places in the project code. Cppcheck - 193, Visual Studio - 116.
We can say that the CppCat and PVS-Studio analyzers detect 4 times more errors than Cppcheck and 6 times more than the analyzer from Visual Studio.
Conclusion
The CppCat and PVS-Studio analyzers proved to be clear leaders in the number of defects found and potential defects.
Additionally
- Evgeny Ryzhkov, Andrey Karpov, Pavel Eremeev, Svyatoslav Razmyslov. “How we compared the code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ).
Have you read the article and immediately want to ask?
Often our articles are asked the same questions. We collected answers to them here: Answers to questions from readers of articles about PVS-Studio and CppCat, version 2014 . Please see the list.
| Unfortunately, we no longer develop or support the CppCat project. You can read about the reasons here . |
