Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation

TL;DR

This paper presents a scalable simulation framework for testing autonomous vehicles by efficiently estimating rare-event probabilities, significantly reducing testing time compared to traditional methods.

Contribution

It introduces an adaptive importance-sampling simulation approach for autonomous vehicle testing, enabling faster and safer evaluation of accident probabilities in complex scenarios.

Findings

Accelerates system evaluation by 2-20 times over naive Monte Carlo methods.

Achieves 10-300 times speedup over real-world testing with parallel processing.

Demonstrates effectiveness on highway scenarios for autonomous vehicle safety assessment.

Abstract

While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the $\textit{de facto}$ evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by $2$-$20$ times over naive Monte Carlo sampling methods and $10$-$300 \mathsf{P}$ times (where $\mathsf{P}$ is the number of processors) over real-world testing.