DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score

TL;DR

DeepMetis is a tool that automatically generates additional test inputs to improve the mutation detection capability of deep learning systems, significantly increasing fault exposure in testing.

Contribution

The paper introduces DeepMetis, a novel search-based input generation method that enhances DL test sets to better detect mutations and faults.

Findings

Increases mutation detection rate by 63% on average.

Effective at exposing unseen mutants, indicating improved test set robustness.

Utilizes multiple model instances to handle non-determinism in training and mutation.

Abstract

Deep Learning (DL) components are routinely integrated into software systems that need to perform complex tasks such as image or natural language processing. The adequacy of the test data used to test such systems can be assessed by their ability to expose artificially injected faults (mutations) that simulate real DL faults. In this paper, we describe an approach to automatically generate new test inputs that can be used to augment the existing test set so that its capability to detect DL mutations increases. Our tool DeepMetis implements a search based input generation strategy. To account for the non-determinism of the training and the mutation processes, our fitness function involves multiple instances of the DL model under test. Experimental results show that \tool is effective at augmenting the given test set, increasing its capability to detect mutants by 63% on average. A leave-one-out experiment shows that the augmented test set is capable of exposing unseen mutants, which simulate the occurrence of yet undetected faults.