Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification

TL;DR

This paper evaluates Bayesian uncertainty quantification methods, MC-Dropout and Deep Ensembles, for early detection of safety-critical misbehaviours in autonomous driving, demonstrating their effectiveness and computational feasibility in simulation benchmarks.

Contribution

It provides a comparative analysis of Bayesian methods for real-time misbehaviour detection, highlighting Deep Ensembles as a highly effective and efficient approach.

Findings

Deep Ensembles detect most misbehaviours with no false alarms.

Both methods outperform autoencoders and attention maps in effectiveness and efficiency.

Uncertainty quantification enables early warnings of unsafe conditions.

Abstract

The automated real-time recognition of unexpected situations plays a crucial role in the safety of autonomous vehicles, especially in unsupported and unpredictable scenarios. This paper evaluates different Bayesian uncertainty quantification methods from the deep learning domain for the anticipatory testing of safety-critical misbehaviours during system-level simulation-based testing. Specifically, we compute uncertainty scores as the vehicle executes, following the intuition that high uncertainty scores are indicative of unsupported runtime conditions that can be used to distinguish safe from failure-inducing driving behaviors. In our study, we conducted an evaluation of the effectiveness and computational overhead associated with two Bayesian uncertainty quantification methods, namely MC- Dropout and Deep Ensembles, for misbehaviour avoidance. Overall, for three benchmarks from the Udacity simulator comprising both out-of-distribution and unsafe conditions introduced via mutation testing, both methods successfully detected a high number of out-of-bounds episodes providing early warnings several seconds in advance, outperforming two state-of-the-art misbehaviour prediction methods based on autoencoders and attention maps in terms of effectiveness and efficiency. Notably, Deep Ensembles detected most misbehaviours without any false alarms and did so even when employing a relatively small number of models, making them computationally feasible for real-time detection. Our findings suggest that incorporating uncertainty quantification methods is a viable approach for building fail-safe mechanisms in deep neural network-based autonomous vehicles.