Property-Based Testing

Property-based testing is a technique I came into contact with that probably changed the way I code the most. It is a powerful tool to describe the properties (/the behavior) of the units software consists of. It greatly reduces the amount of test code that needs to be written, while increasing the coverage of edge cases at the same time. We will start by finding out that we actually have a problem with existing test methods.

The Problem

Achieving a high or even 100% coverage rate with unit tests is already fairly labor-intensive. But even if all the lines, or the branches of code are covered, there still is the possibility of errors being in there. A quick example:

def pop_last(container):
    last = container[-1]
    return (container, last)

def test_pop_last():
    assert pop_last([0,1]) == ([0], 1)

This function gets the last element of a container and returns it in a tuple together with all the other elements in a new container. So far so good. The test case passes. We’re done!

Unfortunately there are edge cases here. The most glaring one is the empty list. Passing an empty list raises ‘IndexError: list index out of range’. Also not all containers provide the pop method. This would namely fail for tuples. In fact, all non-mutable containers would not work.

To tackle this problem equivalence class decomposition should be used to find the “classes” of input that would likely cause (the same) failures. This causes a lot of work in two ways: First, there is a lot of cognitive work involved in thinking through the kinds of inputs that could be given to a function (even ignoring that basically values of all types can be passed in Python). Second, one would have to write a lot of test cases with a lot of repetitive code.

This is where property-based testing aims to help.

How It Works

Property-based testing basically shifts the responsibility of finding relevant test data from the programmer to a framework. The programmer only states that the implementation shall have some properties and lets the framework figure out the rest. Typically, the framework needs to be told the types of the input parameters as well. This sounds fairly abstract now, but it isn’t if we look at an example. The hypothesis library is used for this example. ‘hypothesis’ is an implementation of property-based testing in Python.

import hypothesis
from hypothesis import strategies as hs

def test_pop_last(container):
    if len(container) == 0:
        # typically assert an exception here,
        # but skip for this example
    assert len(pop_last(container)[0]) == len(container) - 1
    assert pop_last(container) == (container[:-1], container[-1])

What happens here? We state the properties that we would expect the implementation to have. The returned container should be one shorter than before, and we explicitly state the split we would expect. The test function is decorated with hypothesis, which will provide us with (100) samples drawn from the “strategy”. It is either a tuple or a list.

It will do so by some thought through code from that framework that tries empty lists, long tuples, zeros in the containers, one element lists and tuples, etc. To summarize, it is acting a lot smarter and working much harder than a programmer that is on a tight schedule and just wishing to get it over with. Hypothesis also offers “shrinking” to find the “easiest” example producing a failure and report it.

From this specifically we see that return (container[:-1], container[-1]) is a much better implementation for all indexable containers (as it creates copies). It will pass this much stricter test. Depending on requirements, the empty list might need to be handled specifically (i.e. by returning ([], None)).

Implementations and Further Reading

Frameworks for property-based testing are available for all major (and some minor) programming languages. I personally have experience with hypothesis for Python and rapidcheck for C++. Both of which I found to be excellent. List of implementations for other languages are available.

I would also recommend reading the article of the creator of hypothesis about what property-based testing means to him. But most important is trying it now and seeing whether it helps your development process or not.

By Tilmann Matthaei

I'm an aspiring software professional looking to share what I learn about reliable software along the way.

I hold a Bachelor's Degree in Applied Computer Science - Digital Media and Games Development and am working in software development since 2018.
I have experience in embedded development (mostly in C++) as well as Continuous Integration and IT Security.

Feel free to contact me via