Does mutation testing improve testing practices?

TL;DR

This study investigates whether mutation testing enhances testing practices by analyzing a large dataset, showing it encourages more comprehensive testing and is linked to real faults, thus potentially improving test quality.

Contribution

The paper provides empirical evidence that mutation testing leads to more extensive test suites and is associated with real faults, filling a gap in understanding its practical benefits.

Findings

Developers using mutation testing write more tests.

Mutants are linked to real faults, indicating practical relevance.

Mutation testing can potentially prevent bugs if used effectively.

Abstract

Various proxy metrics for test quality have been defined in order to guide developers when writing tests. Code coverage is particularly well established in practice, even though the question of how coverage relates to test quality is a matter of ongoing debate. Mutation testing offers a promising alternative: Artificial defects can identify holes in a test suite, and thus provide concrete suggestions for additional tests. Despite the obvious advantages of mutation testing, it is not yet well established in practice. Until recently, mutation testing tools and techniques simply did not scale to complex systems. Although they now do scale, a remaining obstacle is lack of evidence that writing tests for mutants actually improves test quality. In this paper we aim to fill this gap: By analyzing a large dataset of almost 15 million mutants, we investigate how these mutants influenced developers over time, and how these mutants relate to real faults. Our analyses suggest that developers using mutation testing write more tests, and actively improve their test suites with high quality tests such that fewer mutants remain. By analyzing a dataset of past fixes of real high-priority faults, our analyses further provide evidence that mutants are indeed coupled with real faults. In other words, had mutation testing been used for the changes introducing the faults, it would have reported a live mutant that could have prevented the bug.