Categories
Allgemein

Try to Falsify Your Theories Before You Act On Them

Edsger Dijkstra once stated that “[…] program testing can be used very effectively to show the presence of bugs but never to show their absence.” A passing test does not prove you are right. A failing test, however, gives you something solid: an assumption that did not survive contact with reality.

This asymmetry shapes how I think about writing software. Every line of code is based on a theory—about what a function does, how hardware behaves, what the user actually needs. The goal is not to prove yourself correct. It is to find out where you are wrong, as early and as cheaply as possible, before that wrongness gets baked into software you will ship.

Why formulation matters

Most programmers test their assumptions from time to time. You have a hunch something might be slow, so you measure it. You suspect a race condition, so you write a stress test. That is already good practice. But there is a difference between having an assumption in the back of your mind and stating it as an explicit theory. The difference matters in three ways.

First, a properly stated theory opens itself to critique. “This interrupt handler might miss events under load” is a vague worry. You cannot easily disprove it because it does not commit to anything specific. “If the interrupt fires slower than every 200 microseconds, the handler will still process every event” is a theory. It makes a concrete claim. You can write a test that generates interrupts at increasing rates and checks whether the dropout pattern matches the prediction. The precision invites falsification. The vagueness does not.

Second, unstated assumptions feed decision paralysis. When you have not laid out what you believe and how strongly you believe it, every option looks equally risky. You end up stuck because you cannot weigh one concern against another. Formulating “I am 80% confident this message queue blocked because of excessive load in the other process” gives you something to act on. It also tells you exactly where to invest more investigation if that 20% doubt turns out to matter. You move forward with strengthened confidence instead of standing still with diffuse anxiety.

Third, explicit theories make conversations with coworkers more productive. “I chose to apply a moving average before processing this sensor data; I found it to differ by 200 mV between cycles.” That sentence explains the reasoning. A colleague can now engage with the theory directly. Maybe they know something about the sensor or measurement that weakens it. Maybe they can suggest a simpler approach that still satisfies the constraint. What they cannot do is stare at the code and wonder what you were thinking. The theory documents the intent.

The day-to-day cycle

Once the theory is stated, the next step is finding the cheapest way to challenge it. The cost of being wrong rises sharply once code has been integrated, tested on hardware, and deployed. So you want the falsification attempt to happen as early and as lightly as possible.

I ran into this with a popular theory about performance among my coworkers. They have been inlining logic by hand because the theory was: “Function call overhead in this hot path will eat into the timing budget enough to matter.” That is a statement you can check. I collected some timing measurements comparing the inlined version against one with proper function calls. Every other aspect of the code had larger impact than the manual inlining. The measured difference was noise. The theory was false. Thirty minutes saved me from continuing to write harder-to-read code for no reason.

The check does not need to be elaborate. A unit test that exercises a boundary condition is cheap. A small code experiment that runs on the bench is cheap. Asking the engineer next to you “does this match your understanding?” is cheap. The key is that the check must be capable of returning a clear “no” if the theory is wrong. If the test cannot fail, it cannot teach you anything.

I acknowledge that cheap might be a relative term if it comes to safety-critical applications or the implications of a false “no” are otherwise large. Adjust your efforts accordingly.

Testing assumptions about the world

Not all theories are about code behaviour. Some are about the environment the code operates in. These domain assumptions cause just as much waste when they are wrong, but they can be harder to spot because you question them less frequently.

I recently needed to decide whether to keep supporting a particular hardware configuration. The configuration complicated the code with extra branches and special cases. My theory was: “Nobody is using this configuration in the field anymore.” I could have acted on that hunch, or I could have carried the complexity forward out of caution. Instead, I asked the team that manages customer configurations. Ten minutes of conversation confirmed the configuration had been absent from all deployments for years. The theory held, and we removed the code.

That is the same falsification logic applied to the problem domain rather than the implementation. The check was cheap—a conversation instead of a test script—but it served the same purpose. It let reality push back before I committed.

The distinction between essential and accidental complexity runs through this. A system that carries accidental complexity becomes harder to test, harder to modify, and harder to reason about. But you cannot tell which complexity is essential without testing your beliefs about the problem. Expert interviews, searching through field logs, checking with the team that handles customer reports—these are all ways to surface wrong assumptions about the world your code lives in.

Misguided effort

Wrong theories cost you in two ways. The first is direct: you build something based on a false belief, and it breaks. The second is more subtle: you build something based on a belief that is not false, but also not worth spending time on. You optimize a code path that runs once at boot. You guard against a timing condition that cannot occur with your scheduler. You design around a constraint that does not actually constrain anything.

This second category is harder to notice because the code works correctly. It just does not address anything relevant. The only defence is to state the belief clearly enough that you can ask: is this worth my attention right now? “Register spilling is the performance bottleneck” is a theory you can check. But the prior theory — “performance in this code path is currently a problem we need to solve” — also needs checking.

Summing up

The argument is not that you need certainty before you act. It is that the steps should be small, the theories should be written down or spoken aloud, and the checks should be cheap. When each piece of a system rests on a stated and examined belief, the whole holds up better. When something does go wrong, you can trace it back to a specific theory and re-examine it.

The habit builds over time. You start reaching for performance logs, the integration test, or the colleague with domain knowledge before you reach for the editor. You get better at phrasing your hunches in ways that can be shot down. A ten-minute check that saves a week of rework is not a distraction from the real work. It is the real work, done in the right order.

Categories
Allgemein

Software Quality in Academia

Most people, that studied or worked in an academic setting, have come across pieces of software, that are hard to understand and hard to maintain. I just handed in my master thesis, and came across many instances of low quality code during my studies. The most obvious shortcoming, was an almost complete absence of tests. I rarely found tests in the source code related to scientific publications. On a side note, I was not required once in my years of study, to provide any test case whatsoever.

Another big issue is often that academic source code does not have any abstractions. There is a very narrow problem, that is being tackled by the research conducted and reuse of the source code for other (related) problems does not seem to be intended. Interoperability is also often an issue (likely also because of a narrow research scope). Often researchers could easily make a library from their code base, but do not do it.

I acknowledge that there are already good posts on these issues out there, often from people, who are much more involved in academia. I liked the article “On the quality of academic software” by Daniel Lemire a lot. Also, “Why Academic Software Sucks” by Matthias Döring is a good (short) read. However, I would especially like to discuss the incentive structure in more detail, to identify causes and solutions.

Incentives

Academic researchers, be it post-docs, professors, or people working on a bachelor, master or PhD thesis, are judged by their publications or submissions. They are not being judged by the code that supports their research. This incentivizes them to focus on academic writing, instead of writing code. There is in general nothing wrong with that. Academic writing can act as an incredibly accurate and concise documentation to a project. However, if low quality software inhibits further research or reproduction, there is a problem.

Many people working at universities told me that academia is an incredibly competitive field (especially before you become a professor). The pressure to publish breakthrough results at a high pace is immense. In addition to the inherent quality issues arising from such pressure, this also fosters an environment of individualism and distrust. That means that researchers often work on their software completely isolated and have no interest in the success of their peers.

Academic researchers also often receive no economic benefit from providing high quality software. Their research is mostly funded by (public) research grants, and qualitative software requirements are often not a thing.

Possible Solutions

Universities do operate under very specific social, political and economic conditions. Those conditions are unlikely to change abruptly and insofar, everything I can state as a possible solution can likely not get implemented by any individual (contrary to the general claim this blog is based on). However, maybe these points help to identify positive change, where it arises.

It should become the norm in academia to deliver high quality source code and interoperable and robust software. Publications should be judged together with supporting software, as that software is required to reproduce and extend the results of the publication.

Cooperation in academia should be encouraged. Working together on a problem helps to deliver better results. That holds both for delivered software, and the overall results of the research. This can especially come in the form of inter-university open source projects, where an effort is made to maintain a common, well-maintained codebase.

At last, I believe that, where public funds are granted for software-centered research, a high quality open source software should be the result. Such qualitative requirements should be part of research grants.

Categories
Allgemein

Technical Debt

Technical debt sounds scary. From what we know from financial debt, all debt has some inherent risk that it starts consuming all the income/revenue one has. One becomes insolvent and that is not at all what one wants to be. That same thing may happen with technical debt too. If all development efforts are used up by maintenance tasks and no actual progress can be made, a project or organization reaches technical insolvency.

Categories
Allgemein

Property-Based Testing

Property-based testing is a technique I came into contact with that probably changed the way I code the most. It is a powerful tool to describe the properties (/the behavior) of the units software consists of. It greatly reduces the amount of test code that needs to be written, while increasing the coverage of edge cases at the same time. We will start by finding out that we actually have a problem with existing test methods.

Categories
Allgemein

Having a Development Guideline

A software development guideline is a document that determines important rules for the processes, tools, design and architecture used by a team of software developers (programmers, QA folks, product managers…). It is my firm belief that every team of software developers should have such a guideline for the reasons that will be laid out in this article.

Categories
Allgemein

100% Test Coverage

Unit tests are vital in bigger projects. They allow spotting regressions as changes are being made and give the largest share of confidence that everything is in an okay state. But how many unit tests are enough unit tests? In this post I will argue that the optimal coverage rate is 100%.

But let’s start with the bad stuff. Let’s start with the reasons we do not achieve 100% test coverage by default.