TorchProbe: Fuzzing Dynamic Deep Learning Compilers

TL;DR

TorchProbe introduces a fuzzing approach with code transformations to effectively test dynamic features in deep learning compilers, uncovering previously unknown bugs in PyTorch and Triton.

Contribution

The paper presents novel code transformations for fuzzing dynamic deep learning compilers, enabling comprehensive testing of dynamic Python features.

Findings

Identified 20 previously unknown bugs in PyTorch and Triton.

Demonstrated effectiveness of code transformations in testing dynamic features.

Enhanced robustness of deep learning compiler testing methods.

Abstract

Static and dynamic computational graphs represent two distinct approaches to constructing deep learning frameworks. The former prioritizes compiler-based optimizations, while the latter focuses on programmability and user-friendliness. The recent release of PyTorch 2.0, which supports compiling arbitrary deep learning programs in Python, signifies a new direction in the evolution of deep learning infrastructure to incorporate compiler techniques in a more dynamic manner and support more dynamic language features like dynamic control flows and closures. Given PyTorch's seamless integration with Python, its compiler aims to support arbitrary deep learning code written in Python. However, the inherent dynamism of Python poses challenges to the completeness and robustness of the compiler. While recent research has introduced fuzzing to test deep learning compilers, there is still a lack of comprehensive analysis on how to test dynamic features. To address this issue, we propose several code transformations to generate test cases involving dynamic features. These transformations preserve the program's semantics, ensuring that any discrepancy between the transformed and original programs indicates the presence of a bug. Through our approach, we have successfully identified twenty previously unknown bugs in the PyTorch compiler and its underlying tensor compiler Triton.