Scenario-based assessment of automated driving systems: How (not) to parameterize scenarios?

TL;DR

This paper examines how the parameterization of scenarios in automated driving system assessments impacts simulation results, emphasizing the need for careful parameter choices to improve evaluation accuracy and proposing alternative parameterizations.

Contribution

It demonstrates the significant effect of scenario parameterization on simulation outcomes and offers improved parameterization methods aligned with real-world data.

Findings

Parameterization choice greatly influences simulation results.

Alternative parameterizations yield results closer to real-world scenarios.

The impact varies with scenario type, driver model, and evaluation criteria.

Abstract

The development of Automated Driving Systems (ADSs) has advanced significantly. To enable their large-scale deployment, the United Nations Regulation 157 (UN R157) concerning the approval of Automated Lane Keeping Systems (ALKSs) has been approved in 2021. UN R157 requires an activated ALKS to avoid any collisions that are reasonably preventable and proposes a method to distinguish reasonably preventable collisions from unpreventable ones using "the simulated performance of a skilled and attentive human driver". With different driver models, benchmarks are set for ALKSs in three types of scenarios. The three types of scenarios considered in the proposed method in UN R157 assume a certain parameterization without any further consideration. This work investigates the parameterization of these scenarios, showing that the choice of parameterization significantly affects the simulation outcomes. By comparing real-world and parameterized scenarios, we show that the influence of parameterization depends on the scenario type, driver model, and evaluation criterion. Alternative parameterizations are proposed, leading to results that are closer to the non-parameterized scenarios in terms of recall, precision, and F1 score. The study highlights the importance of careful scenario parameterization and suggests improvements to the current UN R157 approach.