A Tale of Two DL Cities: When Library Tests Meet Compiler

TL;DR

This paper introduces OPERA, a testing framework that leverages DL library tests to identify bugs in DL compiler model loading stages, demonstrating improved bug detection and testing efficiency.

Contribution

OPERA extracts domain knowledge from DL library tests to generate diverse test inputs and employs a prioritization strategy, enhancing bug detection in DL compiler model loading stages.

Findings

Detected 170 previously unknown bugs, with 90 confirmed and fixed.

Improved testing efficiency by 11.9% to 47.4% over baseline strategies.

Effectively used tests from multiple DL libraries and compilers.

Abstract

Deep Learning (DL) compilers typically load a DL model and optimize it with intermediate representation.Existing DL compiler testing techniques mainly focus on model optimization stages, but rarely explore bug detection at the model loading stage. Effectively testing the model loading stage requires covering diverse usages of each DL operator from various DL libraries, which shares a common objective with DL library testing, indicating that the embedded knowledge in DL library tests is beneficial for testing the model loading stage of DL compilers. In this work, we propose OPERA to extract such domain knowledge from the test inputs for DL libraries. OPERA constructs diverse tests from the various test inputs for DL libraries (including the test inputs documented in DL libraries and those generated by recent fuzzers). In addition, it incorporates a diversity-based test prioritization strategy to migrate and execute those test inputs that are more likely to detect diverse bugs earlier. We considered three sources of tests in DL libraries for migration and used eight frontends from three DL compilers (e.g., TVM, TensorRT, and OpenVINO) for evaluation. OPERA detected 170 previously unknown bugs in total, 90 of which have been confirmed/fixed by developers, demonstrating the effectiveness of such the migration-based idea. The test prioritization strategy in OPERA improves testing efficiency with migrated tests by 11.9%~47.4% on average compared to general test prioritization strategies.