Deep Learning Library Testing: Definition, Methods and Challenges

TL;DR

This paper reviews the current state of testing methods for deep learning libraries, highlighting their characteristics, challenges, and future research directions to improve the security and reliability of DL systems.

Contribution

It provides a comprehensive overview of DL library testing, analyzing existing methods, their limitations, and outlining future challenges and research opportunities.

Findings

Existing testing methods have notable limitations in effectiveness.

DL library bugs pose significant security and safety risks.

Future research is needed to address testing challenges.

Abstract

In recent years, software systems powered by deep learning (DL) techniques have significantly facilitated people's lives in many aspects. As the backbone of these DL systems, various DL libraries undertake the underlying optimization and computation. However, like traditional software, DL libraries are not immune to bugs, which can pose serious threats to users' personal property and safety. Studying the characteristics of DL libraries, their associated bugs, and the corresponding testing methods is crucial for enhancing the security of DL systems and advancing the widespread application of DL technology. This paper provides an overview of the testing research related to various DL libraries, discusses the strengths and weaknesses of existing methods, and provides guidance and reference for the application of the DL library. This paper first introduces the workflow of DL underlying libraries and the characteristics of three kinds of DL libraries involved, namely DL framework, DL compiler, and DL hardware library. It then provides definitions for DL underlying library bugs and testing. Additionally, this paper summarizes the existing testing methods and tools tailored to these DL libraries separately and analyzes their effectiveness and limitations. It also discusses the existing challenges of DL library testing and outlines potential directions for future research.