MuFF: Stable and Sensitive Post-training Mutation Testing for Deep Learning

TL;DR

MuFF is a new post-training mutation testing method for deep learning that produces stable, sensitive, and killable mutants efficiently, improving upon existing techniques in stability and sensitivity while maintaining high speed.

Contribution

MuFF introduces a novel automated stability check and two mutation operators, enhancing mutant stability and sensitivity in post-training DL mutation testing.

Findings

MuFF generates mutants with 60% and 25% higher sensitivity than DeepMutation++ and DeepCrime.

MuFF produces more stable mutants than DeepMutation++, different from DeepCrime.

MuFF is 61 times faster than DeepCrime in mutant generation.

Abstract

Rapid adoptions of Deep Learning (DL) in a broad range of fields led to the development of specialised testing techniques for DL systems, including DL mutation testing. However, existing post-training DL mutation techniques often generate unstable mutants across multiple training repetitions and multiple applications of the same mutation operator. Additionally, while extremely efficient, they generate mutants without taking into account the mutants' sensitivity and killability, resulting in a large number of ineffective mutants compared to pre-training mutants. In this paper, we present a new efficient post-training DL mutation technique, named MuFF, designed to ensure the stability of the mutants and capable of generating killable and sensitive mutants. MuFF implements an automated stability check and introduces two mutation operators, named weight and neuron inhibitors. Our extensive empirical experiments show that MuFF generates mutants with 60%pt and 25%pt higher sensitivity compared to DeepMutation++ and DeepCrime, respectively, while also producing mutants that are more stable than those of DeepMutation++ and different from the mutants of DeepCrime. Moreover, MuFF preserves the benefits of the post-training mutation technique, being 61 times faster than DeepCrime in generating mutants.