Exposing Previously Undetectable Faults in Deep Neural Networks

TL;DR

This paper presents a novel testing method for deep neural networks that uses generative models to create diverse high-level feature variations, enabling detection of previously undetectable faults.

Contribution

It introduces a new DNN testing approach leveraging generative models to find faults beyond the scope of existing methods.

Findings

Detects deliberately injected faults in DNNs

Identifies new faults in state-of-the-art DNNs

Outperforms existing testing methods in fault detection

Abstract

Existing methods for testing DNNs solve the oracle problem by constraining the raw features (e.g. image pixel values) to be within a small distance of a dataset example for which the desired DNN output is known. But this limits the kinds of faults these approaches are able to detect. In this paper, we introduce a novel DNN testing method that is able to find faults in DNNs that other methods cannot. The crux is that, by leveraging generative machine learning, we can generate fresh test inputs that vary in their high-level features (for images, these include object shape, location, texture, and colour). We demonstrate that our approach is capable of detecting deliberately injected faults as well as new faults in state-of-the-art DNNs, and that in both cases, existing methods are unable to find these faults.