Code Coverage - I want to believe

    The developer must know his tools! Knowledge of tools increases productivity, efficiency, productivity , developer potentiality ! I can not program without R #!

    Phrases of this kind can be heard from completely different people: development fanatics, sellers of various utilities, users of convenient tools. My manager also hears them when I want to experiment with something new.

    True, the instruction manual for the tool usually does not contain the “Contraindications” section; situations where it is NOT worth using the utility are not indicated. Meanwhile, such a section could save tons of time for failed experiments.

    Today I am throwing stones at Code Coverage (CC). A fairly useful metric, under which are several poorly documentedrake.

    "There is a lie, there is a blatant lie
    there is statistics."

    The creators of SonarQube perfectly understand this, not without reason that SonarQube has a dozen CC metrics . I will interrupt statistics using CC from DevExpress, there is only one metric.

    Problem 1. A typical test is wrong. Test the method with a bunch of argument checks:

    public static Hash Example1_IfThen(int arg)
            {
                if(arg == 0) { throw new ArgumentException("0 argument"); }
                if (arg == 1) { throw new ArgumentException("1 argument"); }
                if (arg == 2) { throw new ArgumentException("2 argument"); }
                return new Hash(new Hash(arg + 42) + 13);
            }
    ... 
            [TestMethod, ExpectedException(typeof(ArgumentException))]
            public void Example1_IfThen_0()
                { Program.Example1_IfThen(0); }
            [TestMethod, ExpectedException(typeof(ArgumentException))]
            public void Example1_IfThen_2()
                { Program.Example1_IfThen(2); }
            [TestMethod, ExpectedException(typeof(ArgumentException))]
            public void Example1_IfThen_1()
                { Program.Example1_IfThen(1); }
    

    The method is 83% covered in tests, which is usually enough for an auto build . There is nothing to argue about technically, most of the code is covered by tests, but the main scenario is not affected by tests. The tests cover the simplest part of the code, not the most important.

    Problem 2. We measure the actual code, instead of the necessary.

        public static int Example2_IncompleteLogic(IEnumerable arg)
        {
            return arg
                .Where(elem => elem != 2)
                .Where(elem => elem != 3)
                .Count();
        }
        ...
        [TestMethod]
        public void OneTestToCoverAll()
        {
            // Arrange - collection with one element '4'
            var arg = new List { 4 };
            // Act
            var result = Program.Example2_IncompleteLogic(arg);
            // Assert
            Assert.AreEqual(1, result);
        }
    

    The test method does not contain a check for a null argument, but the coverage is 100%. Sometimes people forget: Code Coverage is a code coverage metric, not a requirements closure metric; if the method lacks logic (the method is not complicated enough to solve its problem) - CC will not show this.

    100% coverage does not guarantee the performance of the program . Bringing to the point of absurdity: the empty method is elementarily covered by 100%. The non-empty method is 100% tested without Assert.

    Problem 3. Optimism. A slightly different manifestation of the previous problem. As you can see, one test covers 100% of the code. Let's try to rewrite our method, getting rid of LINQ (to improve performance).

    var result = 0;
    foreach(var elem in arg)
    {
        if(elem == 2 || elem == 3) { continue; }
        else { result++; }
    }
    return result;

    We get only 73% coverage. Functionality has not changed, the metric has fallen. Not only that, 100% coverage does not guarantee the operability of the program, these 100% can be fake . Conclusion: LINQ - g ** but CC results may be overestimated, try to check coverage in the editor.

    Side observation: in this case, we can just have a jamb in the technical implementation, in theory the anonymous method elem => elem! = 2 can be replaced by elem => if (elem! = 2) return true else return false; , which will fix the coverage of the original method up to 73%. True, this approach will require the complication of the convenient UI now.

    Corollary: The tool used may not have all the desired functionality. A trivial thing, no less true.

    Problem 4. Transfer of responsibility.

    public static void MakeEverythingWell()
    {
        OuterLib.MakeEverythingWell();
    } 

    The method is 100% covered in one test. Moreover, the coverage of the OuterLib library lies with the conscience of the person who added it. Or updated. Three years ago, before the introduction of CC. Before dismissal.
    We have to state the fact again: not only do 100% of the coverage do not guarantee the operability of the program, these 100% can be fake .

    In addition to purely code points, there are several claims specifically to processing CC results


    Claim 0, known to all. 100% coverage. No 100% coverage - no build approval. The problem is that it’s relatively easy to get the first percent of coverage, but the last ones ... Especially when part of the code is generated. Or unattainable (since it was created for Vasya, who will use it in two days). Or just theoretically achievable, and an example to select \ calculate a couple of weeks (this happens when working with mathematics). In short, most teams (of those who generally integrate CC into CI) stop at 60 \ 70 \ 80 percent of the required coverage.

    Claim 1, controversial. Dead code coverage. In my memory, a similar problem was especially pronounced during the verification of Mirandcolleagues from PVS. The comments are quite emotional, but part of the debate concerned the dead code: some of the diagnostics found pointed to (abandoned) plugins, but not to the core .

    The question is: do you need CodeCoverage for dead code ? On the one hand, dead code is a problem, and attention is encouraged. On the other hand, dead code does not affect production, so is it worth it to allow it to influence the CC metric?

    Claim 2. The importance of the code.Extension of the problem 1. There are two notable controllers in my project: “payment” and “negotiation”. “Payment” is critical for the client, and I completely agree with the requirement of “80% coverage”, while 1.5 anonymus use the “negotiation”. In year. And she has not changed for two years. Question: why write tests for half dead functionality? Only to get an 80% badge for auto assembly approval ?

    Claim 3, impossible. Metric as an achievement. This is when no one checks what exactly is covered. Remember the stories about paying for lines of code? I have heard about people who created unnecessary code for better coverage.

    Claim 4. Metric "for free".When the management drops the requirement “cover the code by 80%”, and the developers meekly agree. The project is one-time. Or a prototype. Or a deadline on the nose. Or there’s a hefty pasta legacy monster without a single test.

    Covering code with tests takes time! If the coating is also measured, the time for tests may increase (although it may fall). So if the team did not manage to deliver the project on time, but reached 80% of the coverage, then the fault can be shared between the management and the developers. The question of the dividing line of guilt should not be raised, for holivar.

    At the end. Once again, I note: SS is a useful metric, albeit with surprises. It really helps with code control, if there is no blind desire for numbers in reports.

    Also popular now: