100% Test Coverage

Unit tests are vital in bigger projects. They allow spotting regressions as changes are being made and give the largest share of confidence that everything is in an okay state. But how many unit tests are enough unit tests? In this post I will argue that the optimal coverage rate is 100%.

But let’s start with the bad stuff. Let’s start with the reasons we do not achieve 100% test coverage by default.

The Cost

Let’s consider the following Python function:

def is_even(number: int) -> bool:
    return (number % 2) == 0

It’s a fairly simple one-liner. One could probably be pretty sure that it’s implemented correctly by just looking at it. But let’s write some tests that could be run by pytest.

def test_is_even():
    assert is_even(0)
    assert not is_even(1)
    assert is_even(2)
    assert not is_even(-1)
    assert is_even(-2)

So we now have a test and a function and that function has 100% line and branch coverage through the test. That’s nice, but we also wrote 5 lines of code to test that one-liner.

Many developers shy away from writing tests for simple functions like this for that reason alone. In my experience the test code base will become around twice as big as the production code base if 100% of the functions in there have a test. That’s a lot of code to write, but you will thank yourself later because of the benefits.

The Benefits

After this initial effort, every time we need to revisit a function we do not have to think about its correctness. We just invoke the test runner (in this case pytest) and are now sure that every function in our code base that has a test attached is correct. At least to the extent that we specified its expected behavior through tests. Wouldn’t it be nice if we know that 100% of the functions we have in our code base are correct? If only the way we integrate them might be at fault if something goes wrong?

This might not seem especially impressive at first, but as the code base grows, we want to add abstraction, over abstraction, over abstraction and not think about such small functions anymore.

It might be called from many places in the code and if something was wrong with this function you should probably check the places where it was called from as well. Just to find out why stuff didn’t break there even though it used an incorrectly implemented function.

It also helps with refactoring and restructuring if every time a change is made at least one test case goes red and is waiting to get its expectations updated. And if a coworker needs to use a function, he or she can just have a look at the test case and has a neat example of how to use it and what can be expected.

Overall I would summarize the benefit of having 100% unit test coverage as following: When working with existing code, you don’t think anymore, you just use it!

Tips for achieving 100%

First of, it is important to measure your coverage. In Python 3 I usually use tox as a wrapper for my test cases and have a tox.ini file in the repository that looks something like this.

[tox]
envlist = py38, coverage

[testenv]
requires =
    pytest
commands =
    pytest

[testenv:coverage]
requires =
    pytest
    coverage
commands =
    coverage run --source={envsitepackagesdir}/mypackage --omit __init__.py -m pytest
    coverage report --show-missing --fail-under=100

In Python 3 it is especially easy to measure coverage, because the interpreter keeps track of what was executed. In compiled languages like C++ it is harder but tools like gcov allow you to determine the test coverage as well.

The second tip would be to integrate this coverage analysis in your CI system if you have one. Getting a red circle, if you didn’t achieve the test coverage you wanted, goes a long way.

The third and last tip is about mindset. Think about your future self that will benefit from the tests you write today and just do it. Even if deadlines seem tight and the management is expecting results, think about the place where the company could be in two years from now. Would the company rather benefit from a well-maintained code base that still allows making changes quickly or from that one customer two years ago that got their feature implemented two days earlier?