Leveraging Propagated Infection to Crossfire Mutants

TL;DR

This paper introduces a memory-state analysis and a theoretical model to optimize assertion amplification in mutation testing, significantly increasing mutant detection efficiency by crossfiring multiple mutants with fewer assertions.

Contribution

It presents a novel technique combining memory-state analysis and a theoretical model to enhance assertion amplification for killing multiple mutants efficiently.

Findings

Killed all detectable surviving mutants with only 1.1% of assertion candidates.

Amplified tests increased mutant detection by 6 times on average.

Theoretical model describes crossfiring opportunities at various granularities.

Abstract

Mutation testing was proposed to identify weaknesses in test suites by repeatedly generating artificially faulty versions of the software (mutants) and determining if the test suite is sufficient to detect them (kill them). When the tests are insufficient, each surviving mutant provides an opportunity to improve the test suite. We conducted a study and found that many such surviving mutants (up to 84% for the subjects of our study) are detectable by simply augmenting existing tests with additional assertions, or assertion amplification. Moreover, we find that many of these mutants are detectable by multiple existing tests, giving developers options for how to detect them. To help with these challenges, we created a technique that performs memory-state analysis to identify candidate assertions that developers can use to detect the surviving mutants. Additionally, we build upon prior research that identifies ``crossfiring'' opportunities -- tests that coincidentally kill multiple mutants. To this end, we developed a theoretical model that describes the varying granularities that crossfiring can occur in the existing test suite, which provide opportunities and options for how to kill surviving mutants. We operationalize this model to an accompanying technique that optimizes the assertion amplification of the existing tests to crossfire multiple mutants with fewer added assertions, optionally concentrated within fewer tests. Our experiments show that we can kill all surviving mutants that are detectable with existing test data with only 1.1% of the identified assertion candidates, and increasing by a factor of 6x, on average, the number of killed mutants from amplified tests, over tests that do not crossfire.