Finding Your Missing Tests With Mutation Testing

Have you ever carefully crafted a set of tests, only to later discover that subtle issues still slipped through? Or maybe you’ve wondered if your tests are truly validating the important behaviours, especially in those tricky edge cases. We can sometimes be lulled into a false sense of security: if our tests pass, our code must be solid. But what if there was a way to find those missing edge cases and evaluate the quality of our tests?

Why Code Coverage isn’t enough

Code coverage is a well known, and much discussed, metric when it comes to software development: it can provide a degree of security by pointing you at code your tests don’t exercise, which in turn could allow you to delete dead code (for more on this, see James’s blog: https://technology.lmax.com/posts/coverage-can-only-show-you-what-to-delete/). While it can be handy, it is not perfect, and certainly isn’t the only answer to ensure completeness or quality in our testing.

Coverage can tell us what code paths we aren’t going down, this can help fill in gaps in testing or allow us to delete dead code. If that is not complete, what are we missing? For a start, 100% code coverage can mask untested behaviours. For a basic example, consider:

 public void doSomething(int x) {
    if (x >= 5) {
        // do something
    } else {
        // do something else 
    }
}

An easy path to 100% coverage here would be two tests: one with an input of 6 and the other an input of 4. This is a well known pitfall of code coverage – although we’ve achieved 100% coverage, we are missing a test: what about the case of x == 5? That leaves untested behaviour - what can help us identify these gaps? Enter mutation testing.

Mutation Testing

Mutation testing works by introducing small modifications – “mutations” – to your code and then running your test suite to see if it detects these changes. If a test fails, the mutant code is “killed”; if all tests pass, the mutant “survives”. Surviving mutations can indicate gaps in your test suite – places where you might be missing tests or where assertions could be stronger.

Let’s revisit the previous example. One possible mutation would be to remove the = from our if condition, leaving us with:

    if (x > 5) {
      ...
    }

What happens when our test suite is run with this new code? Well, given our tests weren’t exercising the edge case of x == 5, the removal of = won’t matter and the tests will pass, leaving us with a surviving mutant. This points to a missing test case and allows us to add a test with x = 5 to fill in the gap.

Other examples for mutation operators include removing void method calls, swapping out arithmetic operations (i.e. x + y becomes x – y), or negating conditionals. Each mutation type targets potential weaknesses or untested scenarios in the code, helping to highlight missing or insufficient tests.

Does a survival always indicate a missing test?

A surviving mutant doesn’t always mean there’s a missing test case. Often, it can point to other issues, such as weaknesses in existing assertions. For example, imagine a mutation that removes a void method call intended to update an object’s state. If the test doesn’t assert the expected changes in that object’s fields after calling the method, the mutation will survive. In this case, the problem isn’t a missing test – rather the issues lies with the assertion.

Another reason for a surviving mutant is untested code. If parts of your code are not exercised by any tests but are still mutated, these mutations will survive. For code that has been written using TDD (Test-Driven Development) this is not expected, and any unused code may be valid to delete. For non-TDD code, however, it may indicate entire missing tests rather than just edge cases.

So, a surviving mutant may suggest a missing test, but it could also reveal weak assertions or dead code. This allows you to enhance your test suite without necessarily adding new tests, by either strengthening existing assertions or removing unused code.

Conclusion

Mutation testing can help you find your missing tests, providing an extra degree of security that goes beyond code coverage. By making subtle changes to your code, it actively exposes weaknesses in the test suite – either helping identify incomplete assertions, or missing tests. It can also help you find dead code, although using test coverage is a quicker route to this particular salvation.

What does this mean for you? If you are a code coverage disciple, mutation testing will help you to avoid some of the common pitfalls that can arise when using coverage as part of your pipeline or development process. For teams that prefer instead to rely on TDD to guide the way, mutation testing offers an additional way of improving the quality of your tests and making sure you haven’t missed any edge cases.

• Recently finished a new feature or library? Mutation testing helps catch untested edge cases.
• Concerned about whether your assertions thoroughly verify state changes? Mutation testing can guide you in improving your tests.