The Threat to the Validity of Predictive Mutation Testing: The Impact of Uncovered Mutants

TL;DR

This paper reveals that uncovered mutants significantly distort predictive mutation testing results and proposes a combined machine learning approach to improve its accuracy, demonstrating notable performance decline in previous methods and offering a more reliable alternative.

Contribution

It identifies the impact of uncovered mutants on PMT validity and introduces a novel combined machine learning approach to enhance PMT accuracy in cross-project settings.

Findings

Uncovered mutants inflate PMT results and cause performance decline.

PMT's AUC drops from 0.83 to 0.51 when considering uncovered mutants.

The proposed approach achieves an average AUC of 0.61, improving previous results.

Abstract

Predictive Mutation Testing (PMT) is a technique to predict whether a mutant will be killed by using machine learning approaches. Researchers have proposed various machine learning methods for PMT under the cross-project setting. However, they did not consider the impact of uncovered mutants. A mutant is uncovered if the statement on which the mutant is generated is not executed by any test cases. We show that uncovered mutants inflate previous PMT results. Moreover, we aim at proposing an alternative approach to improve PMT and suggesting a different interpretation for cross-project PMT. We replicated the previous PMT research. We also proposed an approach based on the combination of Random Forest and Gradient Boosting to improve the PMT results. We empirically evaluated our approach on the same 654 Java projects provided by the previous PMT literature. Our results indicate that the performance of PMT drastically decreases in terms of AUC from 0.83 to 0.51. Furthermore, PMT performs worse than random guesses on 27% of the test projects. The proposed approach improves the PMT results by achieving the average AUC value of 0.61.