Full code coverage

Whether it is necessary to do full code coverage with tests is a rather frequent and controversial topic when discussing unit testing. Although most developers are inclined to believe that it is not necessary to do it, that it is inefficient and useless, I have the opposite opinion (at least when developing in Python). In this article I will give an example of how to do full coverage of the code, and describe the disadvantages and advantages of full coverage based on my development experience.

Nose testing tool

For unit testing and statistics collection we use nose . Its advantages in comparison with other means:

• No need to write additional code for binding unit tests
• Built-in metric tools, in particular for calculating coverage percentage
• Python 3 compatible (py3k brunch on google code )

Installing nose should not cause problems - it is installed via easy_install, it is in most Linux repositories, or it can simply be installed from source. For Python 3, you need to make a clone of the py3k branch and install from source.

Initial code example

The factorial calculation will be tested:

#!/usr/bin/env python                                                           
import operator

def factorial(n):
    if n < 0:
        raise ValueError("Factorial can't be calculated for negative numbers.")
    if type(n) is float or type(n) is complex:
        raise TypeError("Factorial doesn't use Gamma function.")
    if n == 0:
        return 1
    return reduce(operator.mul, range(1, n + 1))

if __name__ == '__main__':
    n = input('Enter the positive number: ')
    print '{0}! = {1}'.format(n, factorial(int(n)))


The code works only on Python 2.6 and is not compatible with Python 3. The code is saved in the main.py file .

Unit tests

Let's start with simple tests:

import unittest
from main import factorial

class TestFactorial(unittest.TestCase):

    def test_calculation(self):
        self.assertEqual(720, factorial(6))

    def test_negative(self):
        self.assertRaises(ValueError, factorial, -1)

    def test_float(self):
        self.assertRaises(TypeError, factorial, 1.25)

    def test_zero(self):
        self.assertEqual(1, factorial(0))


These tests only test functionality. Code coverage - 83%: Add another class for 100% coverage:
$ nosetests --with-coverage --cover-erase
....
Name Stmts Exec Cover Missing
-------------------------------------
main 12 10 83% 16-17
----------------------------------------------------------------------
Ran 4 tests in 0.021s

OK


class TestMain(unittest.TestCase):

    class FakeStream:

        def __init__(self):
            self.msgs = []

        def write(self, msg):
            self.msgs.append(msg)

        def readline(self):
            return '5'

    def test_use_case(self):
        fake_stream = self.FakeStream()
        try:
            sys.stdin = sys.stdout = fake_stream
            execfile('main.py', {'__name__': '__main__'})
            self.assertEqual('5! = 120', fake_stream.msgs[1])
        finally:
            sys.stdin = sys.__stdin__
            sys.stdout = sys.__stdout__


Now the code is completely covered by tests:
$ nosetests --with-coverage --cover-erase
.....
Name Stmts Exec Cover Missing
-------------------------------------
main 12 12 100%
----------------------------------------------------------------------
Ran 5 tests in 0.032s

OK


conclusions

Now, based on real code, we can draw some conclusions:

• First and foremost , the full coverage of the code does not provide a complete check of the program’s functionality and does not guarantee its operability. In this example, there were no tests to verify the complex type of the argument, although full coverage was provided.
• You can completely cover the code, at least in Python. Yes, you need to operate with built-in functions and know how certain mechanisms work, but this is real, and it has become even easier in Python 3.
• Python is a dynamically typed programming language, and unit testing helps you do type checking. With full coverage, the likelihood that typing is correctly followed throughout the program is much higher.
• Full coverage helps when changing the API of the libraries used and when changing the programming language itself (see the example for Python 3 below). Because it is guaranteed that every line of code is called, all inconsistencies in the code and API will be detected.
• And as a consequence of the previous paragraph, full coverage helps to test the code. For example, when working on a production system, before software integration, you can first test it. Often, normal debugging is not possible (say, if there are no rights on the remote system, and the administrator is engaged in everything), and unit tests will help to understand where the problem is.

Adaptation for Python 3

Using an adaptation for Python 3, I want to show how full coverage of the code helps in the work. So, first we just run the program under Python 3 and a syntax error is thrown: Correct:
$ python3 main.py
File "main.py", line 17
print '{0}! = {1}'.format(n, factorial(int(n)))
^
SyntaxError: invalid syntax


#!/usr/bin/env python                                                                                                                                       
import operator

def factorial(n):
    if n < 0:
        raise ValueError("Factorial can't be calculated for negative numbers.")
    if type(n) is float or type(n) is complex:
        raise TypeError("Factorial doesn't use Gamma function.")
    if n == 0:
        return 1
    return reduce(operator.mul, range(1, n + 1))

if __name__ == '__main__':
    n = input('Enter the positive number: ')
    print('{0}! = {1}'.format(n, factorial(int(n))))


Now the program can be run: Does this mean that the program is working? Not! It is operational only until reduce is called, which the tests show us: In this example, all this could be detected by manual testing. However, on large projects, only unit testing will help detect this kind of error. And only full coverage of the code can guarantee that almost all inconsistencies in the code and API have been resolved. Well, actually, the working code is fully compatible between Python 2.6 and Python 3:
$ python3 main.py
Enter the positive number: 0
0! = 1


$ nosetests3
E...E
======================================================================
ERROR: test_calculation (tests.TestFactorial)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/nuald/workspace/factorial/tests.py", line 9, in test_calculation
self.assertEqual(720, factorial(6))
File "/home/nuald/workspace/factorial/main.py", line 12, in factorial
return reduce(operator.mul, range(1, n + 1))
NameError: global name 'reduce' is not defined

======================================================================
ERROR: test_use_case (tests.TestMain)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/nuald/workspace/factorial/tests.py", line 38, in test_use_case
execfile('main.py', {'__name__': '__main__'})
NameError: global name 'execfile' is not defined

----------------------------------------------------------------------
Ran 5 tests in 0.010s

FAILED (errors=2)


#!/usr/bin/env python                                                           
import operator
from functools import reduce

def factorial(n):
    if n < 0:
        raise ValueError("Factorial can't be calculated for negative numbers.")
    if type(n) is float or type(n) is complex:
        raise TypeError("Factorial doesn't use Gamma function.")
    if n == 0:
        return 1
    return reduce(operator.mul, range(1, n + 1))

if __name__ == '__main__':
    n = input('Enter the positive number: ')
    print('{0}! = {1}'.format(n, factorial(int(n))))


import sys
import unittest
from main import factorial

class TestFactorial(unittest.TestCase):

    def test_calculation(self):
        self.assertEqual(720, factorial(6))

    def test_negative(self):
        self.assertRaises(ValueError, factorial, -1)

    def test_float(self):
        self.assertRaises(TypeError, factorial, 1.25)

    def test_zero(self):
        self.assertEqual(1, factorial(0))

class TestMain(unittest.TestCase):

    class FakeStream:

        def __init__(self):
            self.msgs = []

        def write(self, msg):
            self.msgs.append(msg)

        def readline(self):
            return '5'

    def test_use_case(self):
        fake_stream = self.FakeStream()
        try:
            sys.stdin = sys.stdout = fake_stream
            obj_code = compile(open('main.py').read(), 'main.py', 'exec')
            exec(obj_code, {'__name__': '__main__'})
            self.assertEqual('5! = 120', fake_stream.msgs[1])
        finally:
            sys.stdin = sys.__stdin__
            sys.stdout = sys.__stdout__


Tests show the full coverage and performance of the program under different versions of Python:
$ nosetests --with-coverage --cover-erase
.....
Name Stmts Exec Cover Missing
-------------------------------------
main 13 13 100%
----------------------------------------------------------------------
Ran 5 tests in 0.038s

OK
$ nosetests3 --with-coverage --cover-erase
.....
Name Stmts Miss Cover Missing
-------------------------------------
main 13 0 100%
----------------------------------------------------------------------
Ran 5 tests in 0.018s

OK


Conclusion

Full code coverage is not a panacea that can protect against program errors. However, this is a tool that you need to know and use. There are many advantages to full coverage, and there is essentially only one drawback - the time and resources required to write tests. But the more you write tests, the easier they will be given to you in the future. For more than a year now, in our projects, we have provided 100% coverage of the code, and although there were a lot of problems in the beginning, now covering the code completely is not a problem at all, because all methods have been worked out and all the necessary packages have been written. There is no magic here (although you will have to work with Python magic), and you just need to start.
PS Full coverage has another advantage, which is not entirely unambiguous, but undoubtedly important for those who consider themselves to be professional - it makes you climb inside Python and understand how it works. This kind of knowledge is useful to everyone, especially library developers.