DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks

TL;DR

DeepEvolution is a search-based testing method for deep neural networks that enhances test case diversity, improves coverage, and detects corner cases and defects more effectively than existing fuzzing techniques.

Contribution

It introduces a novel metaheuristic search approach for DL testing, addressing diversity limitations of prior methods and demonstrating superior effectiveness in real-world models.

Findings

Significantly increases neuronal coverage in test cases.

Successfully identifies corner-case behaviors in DL models.

Outperforms Tensorfuzz in detecting defects during model quantization.

Abstract

The increasing inclusion of Deep Learning (DL) models in safety-critical systems such as autonomous vehicles have led to the development of multiple model-based DL testing techniques. One common denominator of these testing techniques is the automated generation of test cases, e.g., new inputs transformed from the original training data with the aim to optimize some test adequacy criteria. So far, the effectiveness of these approaches has been hindered by their reliance on random fuzzing or transformations that do not always produce test cases with a good diversity. To overcome these limitations, we propose, DeepEvolution, a novel search-based approach for testing DL models that relies on metaheuristics to ensure a maximum diversity in generated test cases. We assess the effectiveness of DeepEvolution in testing computer-vision DL models and found that it significantly increases the neuronal coverage of generated test cases. Moreover, using DeepEvolution, we could successfully find several corner-case behaviors. Finally, DeepEvolution outperformed Tensorfuzz (a coverage-guided fuzzing tool developed at Google Brain) in detecting latent defects introduced during the quantization of the models. These results suggest that search-based approaches can help build effective testing tools for DL systems.