CITADEL: Context Similarity Based Deep Learning Framework Bug Finding

TL;DR

Citadel is a deep learning framework bug detection tool that leverages context similarity to efficiently find new bugs, including performance issues, by generating targeted test cases based on known bug reports.

Contribution

The paper introduces a novel context similarity approach that improves bug detection efficiency and effectiveness in DL frameworks by focusing on similar APIs and automatically generating test cases.

Findings

Detected 58 and 66 API bugs on PyTorch and TensorFlow.

Identified 13 performance bugs that existing tools missed.

Generated test cases that triggered bugs 35.40% more effectively than prior methods.

Abstract

With the application of deep learning technology, tools of DL framework testing are in high demand. Existing DL framework testing tools have limited coverage of bug types. For example, they lack the capability of effectively finding performance bugs, which are critical for DL models regarding performance, economics, and the environment. Moreover, existing tools are inefficient, generating hundreds of test cases with few trigger bugs. In this paper, we propose Citadel, a method that accelerates bug finding in terms of efficiency and effectiveness. We observe that many DL framework bugs are similar due to the similarity of operators and algorithms belonging to the same family. Orthogonal to existing bug-finding tools, Citadel aims to find new bugs that are similar to reported ones that have known test oracles. Citadel defines context similarity to measure the similarity of DL framework API pairs and automatically generates test cases with oracles for APIs that are similar to the problematic APIs in existing bug reports. Citadel effectively detects 58 and 66 API bugs on PyTorch and TensorFlow (excluding those rejected by developers or duplicates of prior reports), many of which, e.g., 13 performance bugs, cannot be detected by existing tools. Moreover, 35.40% of test cases generated by Citadel can trigger bugs significantly transcending the state-of-the-art method (3.90%).