DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation

TL;DR

DevMuT is a novel testing framework for deep learning models that uses developer expertise-based mutation to generate diverse models and detect important defects across the model lifecycle, outperforming existing methods.

Contribution

The paper introduces DevMuT, a new DL framework testing method that incorporates developer expertise into mutation operators to identify critical bugs more effectively.

Findings

Achieves at least 71.68% improvement in model diversity

Detects 117 defects with 63 confirmed and 24 fixed

Outperforms state-of-the-art baselines in defect detection

Abstract

Deep learning (DL) frameworks are the fundamental infrastructure for various DL applications. Framework defects can profoundly cause disastrous accidents, thus requiring sufficient detection. In previous studies, researchers adopt DL models as test inputs combined with mutation to generate more diverse models. Though these studies demonstrate promising results, most detected defects are considered trivial (i.e., either treated as edge cases or ignored by the developers). To identify important bugs that matter to developers, we propose a novel DL framework testing method DevMuT, which generates models by adopting mutation operators and constraints derived from developer expertise. DevMuT simulates developers'common operations in development and detects more diverse defects within more stages of the DL model lifecycle (e.g., model training and inference). We evaluate the performance of DevMuT on three widely used DL frameworks (i.e., PyTorch, JAX, and Mind- Spore) with 29 DL models from nine types of industry tasks. The experiment results show that DevMuT outperforms state-of-the-art baselines: it can achieve at least 71.68% improvement on average in the diversity of generated models and 28.20% improvement on average in the legal rates of generated models. Moreover, DevMuT detects 117 defects, 63 of which are confirmed, 24 are fixed, and eight are of high value confirmed by developers. Finally, DevMuT has been deployed in the MindSpore community since December 2023. These demonstrate the effectiveness of DevMuT in detecting defects that are close to the real scenes and are of concern to developers.