Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling

TL;DR

This paper presents Bayesian adaptive multifidelity sampling (BAMS), a method that efficiently discovers failure scenarios in autonomous vehicles and accurately estimates their adverse event rates, outperforming traditional sampling techniques.

Contribution

The paper introduces BAMS, a novel adaptive Bayesian sampling method that improves failure discovery and rate estimation in autonomous vehicle safety testing.

Findings

BAMS discovers 10 times more issues than MC and IS methods.

Rate estimates from BAMS have 15 and 6 times narrower variances than MC and IS.

BAMS effectively balances exploration and rate estimation in AV safety analysis.

Abstract

Ensuring the safety of autonomous vehicles (AVs) requires both accurate estimation of their performance and efficient discovery of potential failure cases. This paper introduces Bayesian adaptive multifidelity sampling (BAMS), which leverages the power of adaptive Bayesian sampling to achieve efficient discovery while simultaneously estimating the rate of adverse events. BAMS prioritizes exploration of regions with potentially low performance, leading to the identification of novel and critical scenarios that traditional methods might miss. Using real-world AV data we demonstrate that BAMS discovers 10 times as many issues as Monte Carlo (MC) and importance sampling (IS) baselines, while at the same time generating rate estimates with variances 15 and 6 times narrower than MC and IS baselines respectively.