Misbehavior Forecasting for Focused Autonomous Driving Systems Testing

TL;DR

Foresee is a novel simulation-based testing technique for autonomous driving that predicts near misses to efficiently uncover failures, outperforming existing methods in effectiveness and speed.

Contribution

We introduce Foresee, a misbehavior forecasting approach that improves failure detection in autonomous vehicle testing by predicting near misses and guiding local fuzzing.

Findings

Foresee detects significantly more failures than baseline methods.

Foresee is more efficient, being faster than existing predictors.

Combining Foresee with DriveFuzz enhances failure detection by up to 93.94%.

Abstract

Simulation-based testing is the standard practice for assessing the reliability of self-driving cars' software before deployment. Existing bug-finding techniques are either unreliable or expensive. We build on the insight that near misses observed during simulations may point to potential failures. We propose Foresee, a technique that identifies near misses using a misbehavior forecaster that computes possible future states of the ego-vehicle under test. Foresee performs local fuzzing in the neighborhood of each candidate near miss to surface previously unknown failures. In our empirical study, we evaluate the effectiveness of different configurations of Foresee using several scenarios provided in the CARLA simulator on both end-to-end and modular self-driving systems and examine its complementarity with the state-of-the-art fuzzer DriveFuzz. Our results show that Foresee is both more effective and more efficient than the baselines. Foresee exposes 128.70% and 38.09% more failures than a random approach and a state-of-the-art failure predictor while being 2.49x and 1.42x faster, respectively. Moreover, when used in combination with DriveFuzz, Foresee enhances failure detection by up to 93.94%.