Optimization-Aware Test Generation for Deep Learning Compilers

TL;DR

This paper introduces OATest, a novel method for generating optimization-aware computational graphs to test deep learning compilers, leading to the discovery of numerous previously unknown bugs and improved coverage.

Contribution

OATest combines pattern extraction and graph synthesis techniques to effectively generate optimization-aware tests for DL compilers, addressing a key testing challenge.

Findings

Outperforms existing methods in bug detection and code coverage

Uncovered 58 new bugs, with 36 confirmed or fixed by developers

Achieved higher bug detection rate on TVM and ONNXRuntime

Abstract

Deep Learning (DL) compilers have been widely utilized to optimize DL models for efficient deployment across various hardware. Due to their vital role in the DL ecosystem, ensuring their reliability and security is critical. However, existing approaches have limitations in testing optimization stages, which is the core functionality of DL compilers, due to the difficulty in generating optimization-aware tests. In this paper, we proposed OATest, a novel approach for synthesizing optimization-aware computational graphs. The approach combines patterns extracted from documented tests for optimization and incorporates them into seed computational graphs, enabling broader exploration of optimization paths. To guarantee the optimization-awareness of generated graphs, OATest introduces the edges reusing strategy to establish strong connections between patterns and contexts. Additionally, to solve the validity challenge for the generated graphs, OATest employs an auxiliary layers addition strategy to resolve broken constraints. Equipped with two distinct test oracles, OATest applies differential testing to evaluate the two widely used DL compilers (i.e., TVM and ONNXRuntime). Our experimental results show that OATest outperforms the state-of-the-art method by detecting more bugs and achieving higher code coverage in TVM and ONNXRutimes. Additionally, OATest uncovers 58 previously unknown bugs, 36 of which have been confirmed or fixed by developers.