Test Against High-Dimensional Uncertainties: Accelerated Evaluation of Autonomous Vehicles with Deep Importance Sampling

TL;DR

This paper introduces Deep Importance Sampling, a neural network-based method that significantly improves the efficiency of evaluating autonomous vehicle safety under rare, high-dimensional uncertainties, reducing sample size needs and providing more accurate failure rate estimates.

Contribution

The work presents a novel Deep IS framework that enhances importance sampling efficiency for high-dimensional AV evaluation, outperforming existing methods in accuracy and sample reduction.

Findings

Reduces sample size 43 times for 10% error in failure rate estimation.

Achieves over 600 times efficiency boost in high-dimensional traffic sign classification.

Produces less conservative and more precise failure rate estimates.

Abstract

Evaluating the performance of autonomous vehicles (AV) and their complex subsystems to high precision under naturalistic circumstances remains a challenge, especially when failure or dangerous cases are rare. Rarity does not only require an enormous sample size for a naive method to achieve high confidence estimation, but it also causes dangerous underestimation of the true failure rate and it is extremely hard to detect. Meanwhile, the state-of-the-art approach that comes with a correctness guarantee can only compute an upper bound for the failure rate under certain conditions, which could limit its practical uses. In this work, we present Deep Importance Sampling (Deep IS) framework that utilizes a deep neural network to obtain an efficient IS that is on par with the state-of-the-art, capable of reducing the required sample size 43 times smaller than the naive sampling method to achieve 10% relative error and while producing an estimate that is much less conservative. Our high-dimensional experiment estimating the misclassification rate of one of the state-of-the-art traffic sign classifiers further reveals that this efficiency still holds true even when the target is very small, achieving over 600 times efficiency boost. This highlights the potential of Deep IS in providing a precise estimate even against high-dimensional uncertainties.