Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting

TL;DR

This paper compares mutation coverage and branch coverage in an industrial setting, showing mutation coverage detects more test weaknesses with acceptable overhead, encouraging broader adoption in continuous integration.

Contribution

It provides empirical evidence that mutation coverage uncovers additional test weaknesses over branch coverage in industrial projects, addressing adoption barriers.

Findings

Mutation coverage detects more weaknesses than branch coverage.

Mutation coverage has acceptable performance overhead.

Industrial case study supports broader adoption of mutation testing.

Abstract

The state-of-the-practice in software development is driven by constant change fueled by continuous integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is "good enough", and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage and that it is able to do so with an acceptable performance overhead during project build.