On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles

TL;DR

This paper emphasizes the importance of establishing a rigorous statistical foundation for scenario-based testing of autonomous vehicles to improve safety assessment, addressing current gaps and proposing initial models for quantifying failure probabilities.

Contribution

It introduces proof-of-concept models for quantifying failure probabilities and evaluates testing effectiveness, bridging the gap between scenario-based and traditional mile-based testing methods.

Findings

Neither scenario-based nor mile-based testing is universally superior.

Formal reasoning can align synthetic and real-world testing outcomes.

A statistical framework is crucial for defensible safety claims.

Abstract

Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (\textit{pfs}) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims.