Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects

TL;DR

This paper introduces RANUM, a novel approach for ensuring the reliability of deep neural networks against numerical defects by detecting, confirming, and fixing potential issues automatically, outperforming existing methods.

Contribution

RANUM is the first approach to automatically confirm defect feasibility with failure tests and suggest fixes, significantly improving DNN reliability assurance.

Findings

RANUM outperforms state-of-the-art methods in three assurance tasks.

RANUM's fixes are often as good as or better than human fixes.

Extensive experiments on 63 real-world DNN architectures validate effectiveness.

Abstract

With the widespread deployment of deep neural networks (DNNs), ensuring the reliability of DNN-based systems is of great importance. Serious reliability issues such as system failures can be caused by numerical defects, one of the most frequent defects in DNNs. To assure high reliability against numerical defects, in this paper, we propose the RANUM approach including novel techniques for three reliability assurance tasks: detection of potential numerical defects, confirmation of potential-defect feasibility, and suggestion of defect fixes. To the best of our knowledge, RANUM is the first approach that confirms potential-defect feasibility with failure-exhibiting tests and suggests fixes automatically. Extensive experiments on the benchmarks of 63 real-world DNN architectures show that RANUM outperforms state-of-the-art approaches across the three reliability assurance tasks. In addition, when the RANUM-generated fixes are compared with developers' fixes on open-source projects, in 37 out of 40 cases, RANUM-generated fixes are equivalent to or even better than human fixes.