Practical validation of synthetic pre-crash scenarios

TL;DR

This paper presents a Bayesian ROPE-based framework with binning methods for practically validating synthetic pre-crash scenarios against real data, enhancing safety assessment accuracy.

Contribution

It introduces a novel binning-based equivalence testing approach within a Bayesian framework for validating synthetic safety scenarios.

Findings

The framework effectively assesses practical equivalence of synthetic and real datasets.

It provides diagnostic insights into dataset divergences.

Demonstrated on rear-end pre-crash data for automatic emergency braking.

Abstract

The representativeness of synthetic pre-crash scenarios is crucial for assessing the safety impact of Driving Automation Systems through virtual simulations. However, a gap remains in the robust evaluation of synthetic pre-crash scenarios' practical equivalence to their real-world counterparts; that is, whether they are similar enough for the intended assessment purpose. Conventional significance testing is inadequate, as it focuses on detecting differences rather than establishing practical equivalence. This study addresses the research gap by extending our previous work on a Bayesian Region of Practical Equivalence (ROPE)-based equivalence testing framework by introducing a binning-based approach to define appropriate statistics and equivalence criteria. Two binning-based statistics are proposed to measure practically meaningful distributional differences between datasets in the context of safety impact assessment. The framework's applicability is demonstrated through a case study, which tests the practical equivalence of two synthetic rear-end pre-crash datasets with a previously developed reference dataset in the context of the safety impact assessment of an Automatic Emergency Braking system. The results show that the framework provides informative quantitative assessments of practical equivalence as well as diagnostic insights into the divergence of datasets. Although the demonstration focuses on rear-end pre-crash scenarios, the framework is generic and extensible to broader validation contexts, providing an interpretable and principled basis for practical equivalence assessment across diverse synthetic data applications.