New Formulation of DNN Statistical Mutation Killing for Ensuring Monotonicity: A Technical Report

TL;DR

This paper introduces a new Fisher exact test-based formulation for DNN mutation testing that maintains statistical rigor and guarantees monotonicity, addressing limitations of previous methods.

Contribution

It proposes a novel mutation killing criterion for DNNs that ensures monotonicity while preserving statistical testing validity.

Findings

Ensures monotonicity in mutation testing results.

Maintains statistical rigor with Fisher exact test.

Addresses limitations of previous approaches.

Abstract

Mutation testing has emerged as a powerful technique for evaluating the effectiveness of test suites for Deep Neural Networks. Among existing approaches, the statistical mutant killing criterion of DeepCrime has leveraged statistical testing to determine whether a mutant significantly differs from the original model. However, it suffers from a critical limitation: it violates the monotonicity property, meaning that expanding a test set may result in previously killed mutants no longer being classified as killed. In this technical report, we propose a new formulation of statistical mutant killing based on Fisher exact test that preserves the statistical rigour of it while ensuring monotonicity.