Boundary State Generation for Testing and Improvement of Autonomous Driving Systems

TL;DR

This paper introduces GENBO, a novel method for generating boundary driving conditions in simulated environments to test and improve autonomous driving systems by retraining models on challenging scenarios.

Contribution

GENBO is a new test generator that mutates driving conditions at the boundary of model failure, enhancing the dependability of autonomous driving systems.

Findings

Retrained models show up to 3x higher success rates.

Boundary condition generation improves model robustness.

Method effectively identifies challenging driving scenarios.

Abstract

Recent advances in Deep Neural Networks (DNNs) and sensor technologies are enabling autonomous driving systems (ADSs) with an ever-increasing level of autonomy. However, assessing their dependability remains a critical concern. State-of-the-art ADS testing approaches modify the controllable attributes of a simulated driving environment until the ADS misbehaves. In such approaches, environment instances in which the ADS is successful are discarded, despite the possibility that they could contain hidden driving conditions in which the ADS may misbehave. In this paper, we present GENBO (GENerator of BOundary state pairs), a novel test generator for ADS testing. GENBO mutates the driving conditions of the ego vehicle (position, velocity and orientation), collected in a failure-free environment instance, and efficiently generates challenging driving conditions at the behavior boundary (i.e., where the model starts to misbehave) in the same environment instance. We use such boundary conditions to augment the initial training dataset and retrain the DNN model under test. Our evaluation results show that the retrained model has, on average, up to 3x higher success rate on a separate set of evaluation tracks with respect to the original DNN model.