Finding Higher Order Mutants Using Variational Execution

TL;DR

This paper introduces a variational execution method to efficiently identify strongly subsuming higher order mutants (SSHOMs) in mutation testing, outperforming genetic algorithms in speed and completeness.

Contribution

The paper presents a novel variational execution approach for finding SSHOMs, improving speed and reliability over existing genetic algorithms.

Findings

Variational execution finds all SSHOMs in 4.5 seconds.

Genetic algorithm finds 36 of 38 SSHOMs in 50 seconds.

Variational execution outperforms genetic algorithms in speed and completeness.

Abstract

Mutation testing is an effective but time consuming method for gauging the quality of a test suite. It functions by repeatedly making changes, called mutants, to the source code and checking whether the test suite fails (i.e., whether the mutant is killed). Recent work has shown cases in which applying multiple changes, called a higher order mutation, is more difficult to kill than a single change, called a first order mutation. Specifically, a special kind of higher order mutation, called a strongly subsuming higher order mutation (SSHOM), can enable equivalent accuracy in assessing the quality of the test suite with fewer executions of tests. Little is known about these SSHOMs, as they are difficult to find. Our goal in this research is to identify a faster, more reliable method for finding SSHOMs in order to characterize them in the future. We propose an approach based on variational execution to find SSHOMs. Preliminary results indicate that variational execution performs better than the existing genetic algorithm in terms of speed and completeness of results. Out of a set of 33 first order mutations, our variational execution approach finds all 38 SSHOMs in 4.5 seconds, whereas the genetic algorithm only finds 36 of the 38 SSHOMs in 50 seconds.