Search-based DNN Testing and Retraining with GAN-enhanced Simulations

TL;DR

This paper introduces a novel approach combining search-based testing with GAN-generated realistic images to improve DNN testing and retraining in safety-critical applications, outperforming existing methods.

Contribution

It presents a new method that integrates GANs with meta-heuristic search for more effective simulator-based DNN testing and retraining, enhancing test diversity and model performance.

Findings

Outperforms state-of-the-art GAN-based testing solutions.

Generates more diverse images that reveal worst DNN performance.

Leads to significant improvements in DNN accuracy after retraining.

Abstract

In safety-critical systems (e.g., autonomous vehicles and robots), Deep Neural Networks (DNNs) are becoming a key component for computer vision tasks, particularly semantic segmentation. Further, since the DNN behavior cannot be assessed through code inspection and analysis, test automation has become an essential activity to gain confidence in the reliability of DNNs. Unfortunately, state-of-the-art automated testing solutions largely rely on simulators, whose fidelity is always imperfect, thus affecting the validity of test results. To address such limitations, we propose to combine meta-heuristic search, used to explore the input space using simulators, with Generative Adversarial Networks (GANs), to transform the data generated by simulators into realistic input images. Such images can be used both to assess the DNN performance and to retrain the DNN more effectively. We applied our approach to a state-of-the-art DNN performing semantic segmentation and demonstrated that it outperforms a state-of-the-art GAN-based testing solution and several baselines. Specifically, it leads to the largest number of diverse images leading to the worst DNN performance. Further, the images generated with our approach, lead to the highest improvement in DNN performance when used for retraining. In conclusion, we suggest to always integrate GAN components when performing search-driven, simulator-based testing.