Towards Full-Scenario Safety Evaluation of Automated Vehicles: A Volume-Based Method

TL;DR

This paper introduces a volume-based framework for comprehensive safety evaluation of automated vehicles, overcoming data dependency and computational challenges of traditional methods, and enabling precise risk quantification in complex scenarios.

Contribution

It proposes a unified scenario modeling approach and a novel volume-based evaluation method that accurately measures risky scenarios without relying on probability estimates.

Findings

The volume-based method effectively quantifies risky scenarios.

Convexity of safe scenarios in car-following enables exact volume calculation.

Experimental validation shows the method's accuracy and applicability.

Abstract

With the rapid development of automated vehicles (AVs) in recent years, commercially available AVs are increasingly demonstrating high-level automation capabilities. However, most existing AV safety evaluation methods are primarily designed for simple maneuvers such as car-following and lane-changing. While suitable for basic tests, these methods are insufficient for assessing high-level automation functions deployed in more complex environments. First, these methods typically use crash rate as the evaluation metric, whose accuracy heavily depends on the quality and completeness of naturalistic driving environment data used to estimate scenario probabilities. Such data is often difficult and expensive to collect. Second, when applied to diverse scenarios, these methods suffer from the curse of dimensionality, making large-scale evaluation computationally intractable. To address these challenges, this paper proposes a novel framework for full-scenario AV safety evaluation. A unified model is first introduced to standardize the representation of diverse driving scenarios. This modeling approach constrains the dimension of most scenarios to a regular highway setting with three lanes and six surrounding background vehicles, significantly reducing dimensionality. To further avoid the limitations of probability-based method, we propose a volume-based evaluation method that quantifies the proportion of risky scenarios within the entire scenario space. For car-following scenarios, we prove that the set of safe scenarios is convex under specific settings, enabling exact volume computation. Experimental results validate the effectiveness of the proposed volume-based method using both AV behavior models from existing literature and six production AV models calibrated from field-test trajectory data in the Ultra-AV dataset. Code and data will be made publicly available upon acceptance of this paper.