Distribution-Aware Testing of Neural Networks Using Generative Models

TL;DR

This paper introduces a distribution-aware testing approach for neural networks that leverages generative models to generate only valid test inputs, improving test reliability and coverage accuracy.

Contribution

It proposes a novel technique using deep generative models to ensure only valid inputs are generated during DNN testing, addressing a key limitation of prior methods.

Findings

Effectively eliminates invalid test inputs

Increases the number of valid test inputs generated

Provides more accurate test coverage metrics

Abstract

The reliability of software that has a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications being deployed with DNNs. The need for reliability raises a need for rigorous testing of the safety and trustworthiness of these systems. In the last few years, there have been a number of research efforts focused on testing DNNs. However the test generation techniques proposed so far lack a check to determine whether the test inputs they are generating are valid, and thus invalid inputs are produced. To illustrate this situation, we explored three recent DNN testing techniques. Using deep generative model based input validation, we show that all the three techniques generate significant number of invalid test inputs. We further analyzed the test coverage achieved by the test inputs generated by the DNN testing techniques and showed how invalid test inputs can falsely inflate test coverage metrics. To overcome the inclusion of invalid inputs in testing, we propose a technique to incorporate the valid input space of the DNN model under test in the test generation process. Our technique uses a deep generative model-based algorithm to generate only valid inputs. Results of our empirical studies show that our technique is effective in eliminating invalid tests and boosting the number of valid test inputs generated.