DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing

TL;DR

DeepRoad is an unsupervised GAN-based framework that generates realistic diverse driving scenes to test and identify behavioral inconsistencies in autonomous driving DNNs, enhancing safety verification.

Contribution

It introduces a novel GAN-based method for generating accurate, diverse weather conditions for autonomous driving testing, addressing limitations of previous test input quality.

Findings

Detected thousands of behavioral inconsistencies in tested systems.

Generated realistic weather variations including extreme conditions.

Proved effectiveness of DeepRoad in autonomous driving safety testing.

Abstract

While Deep Neural Networks (DNNs) have established the fundamentals of DNN-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To resolve the safety issues of autonomous driving systems, a recent set of testing techniques have been designed to automatically generate test cases, e.g., new input images transformed from the original ones. Unfortunately, many such generated input images often render inferior authenticity, lacking accurate semantic information of the driving scenes and hence compromising the resulting efficacy and reliability. In this paper, we propose DeepRoad, an unsupervised framework to automatically generate large amounts of accurate driving scenes to test the consistency of DNN-based autonomous driving systems across different scenes. In particular, DeepRoad delivers driving scenes with various weather conditions (including those with rather extreme conditions) by applying the Generative Adversarial Networks (GANs) along with the corresponding real-world weather scenes. Moreover, we have implemented DeepRoad to test three well-recognized DNN-based autonomous driving systems. Experimental results demonstrate that DeepRoad can detect thousands of behavioral inconsistencies in these systems.