Deep Learning Framework Testing via Model Mutation: How Far Are We?

TL;DR

This paper evaluates the effectiveness of mutation-based testing for deep learning frameworks, identifying limitations and proposing optimizations that led to discovering new high-priority defects confirmed by developers.

Contribution

It critically analyzes existing mutation testing methods for DL frameworks, highlights their shortcomings, and introduces optimized strategies that successfully detect new critical defects.

Findings

Existing mutation methods have high false positives and limited defect detection.

Optimizations improved defect detection, leading to 7 new defects identified.

Most detected defects were confirmed and addressed by developers.

Abstract

Deep Learning (DL) frameworks are a fundamental component of DL development. Therefore, the detection of DL framework defects is important and challenging. As one of the most widely adopted DL testing techniques, model mutation has recently gained significant attention. In this study, we revisit the defect detection ability of existing mutation-based testing methods and investigate the factors that influence their effectiveness. To begin with, we reviewed existing methods and observed that many of them mutate DL models (e.g., changing their parameters) without any customization, ignoring the unique challenges in framework testing. Another issue with these methods is their limited effectiveness, characterized by a high rate of false positives caused by illegal mutations arising from the use of generic, non-customized mutation operators. Moreover, we tracked the defects identified by these methods and discovered that most of them were ignored by developers. Motivated by these observations, we investigate the effectiveness of existing mutation-based testing methods in detecting important defects that have been authenticated by framework developers. We begin by collecting defect reports from three popular frameworks and classifying them based on framework developers' ratings to build a comprehensive dataset. We then perform an in-depth analysis to uncover valuable insights. Based on our findings, we propose optimization strategies to address the shortcomings of existing approaches. Following these optimizations, we identified seven new defects, four of which were confirmed by developers as high-priority issues, with three resolved. In summary, we identified 39 unique defects across just 23 models, of which 31 were confirmed by developers, and eight have been fixed.