On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering

TL;DR

This paper introduces DEEPMAACC, a clustering-based method that significantly accelerates deep neural network mutation analysis by reducing the number of mutants tested, with acceptable accuracy trade-offs.

Contribution

The paper proposes a novel clustering approach for DNN mutation analysis that improves efficiency while maintaining mutation score accuracy.

Findings

Neuron clustering accelerates analysis by 69.77%.

Mutant clustering accelerates analysis by 35.31%.

Trade-off between speed and mutation score error is demonstrated.

Abstract

Mutation analysis of deep neural networks (DNNs) is a promising method for effective evaluation of test data quality and model robustness, but it can be computationally expensive, especially for large models. To alleviate this, we present DEEPMAACC, a technique and a tool that speeds up DNN mutation analysis through neuron and mutant clustering. DEEPMAACC implements two methods: (1) neuron clustering to reduce the number of generated mutants and (2) mutant clustering to reduce the number of mutants to be tested by selecting representative mutants for testing. Both use hierarchical agglomerative clustering to group neurons and mutants with similar weights, with the goal of improving efficiency while maintaining mutation score. DEEPMAACC has been evaluated on 8 DNN models across 4 popular classification datasets and two DNN architectures. When compared to exhaustive, or vanilla, mutation analysis, the results provide empirical evidence that neuron clustering approach, on average, accelerates mutation analysis by 69.77%, with an average -26.84% error in mutation score. Meanwhile, mutant clustering approach, on average, accelerates mutation analysis by 35.31%, with an average 1.96% error in mutation score. Our results demonstrate that a trade-off can be made between mutation testing speed and mutation score error.