Intersection of PVS-Studio and Cppcheck

    We were repeatedly asked the question of how much the diagnostics of our PVS-Studio analyzer and the Cppcheck analyzer overlap. I decided to write a short article on this topic to quickly answer this question. If very briefly, then they intersect weakly. Only 6% of the total number of errors are found by both analyzers. The article will describe how this number was obtained.

    In the beginning, I wanted to draw a Venn diagram, in the form of beautiful circles. But it turns out that this is a whole task. Excel draw circles without considering their area. And the programs that draw the correct proportional diagrams are paid. So I limited myself to the squares, for which I needed only a calculator for calculations, a pen with paper and the Paint editor.

    Figure 1. Visual display of the number of errors found using the PVS-Studio analyzer and Cppcheck.
    Figure 1. Visual display of the number of errors found using the PVS-Studio analyzer and Cppcheck.

    The square area is proportional to the number of errors found. The gray rectangle displays the number of errors that both analyzers find at the same time.

    Total:
    • Number of errors found with PVS-Studio: 742
    • Number of errors found with Cppcheck: 193
    • In total, the analyzers found 884 unique errors.
    • Number of errors detected by both analyzers: 51
    • Intersection is: 6%.

    The data were obtained as follows. In March 2014, we made a large comparison of four code analyzers: PVS-Studio, CppCat, Cppcheck, Visual Studio:
    The results were severely criticized by some of our readers. But we are sure that most of the criticism is due to the fact that people read brief conclusions, but did not carefully read the article describing the comparison process itself.

    Since the PVS-Studio analyzer performed much better than Cppcheck, some readers decided that we were cheating. In fact, there is no deception. PVS-Studio analyzer is really more powerful than Cppcheck. It is not clear what is unexpected and suspicious in this. Commercial tools are generally better than free peers. High quality comparisons of analyzers, confirmed by the author of Cppcheck. However, we publish an article where we present his letter on the comparison of analyzers, and also answer a number of questions asked by readers after the publication of an article on comparison.

    Back to the intersection for diagnostics. As you can see, the intersection is rather weak, but this is not surprising. Firstly, no one seeks to copy all the diagnostics from another analyzer. The intersection occurs due to those errors whose patterns are obvious and common. Thus, the authors come to the idea of ​​finding such errors independently.

    Secondly, the era of static analyzers is just beginning. There is an incredible amount of error patterns that they can diagnose. Therefore, there are so few intersections. One analyzer relies on errors of one type, the other on others. It is clear that over time, the intersection will gradually increase. But there are so many patterns that this process will be slow. In addition, with the advent of C ++ 11 and C ++ 14, the field for activity is only increasing.
    Unfortunately, we no longer develop or support the CppCat project. You can read about the reasons here .

    Also popular now: