Evaluation Uncertainty in Data-Driven Self-Driving Testing

TL;DR

This paper introduces an efficient bootstrap-based method to quantify the impact of data variability on safety probability estimates in self-driving car evaluations, enhancing reliability assessments.

Contribution

It proposes a novel combination of bootstrap and likelihood ratio techniques to assess input uncertainty in Monte Carlo safety evaluations for autonomous vehicles.

Findings

The method effectively reuses experimental data to evaluate uncertainty.

Application to AV safety demonstrates diagnostic capabilities.

Approach reduces implementation costs for uncertainty quantification.

Abstract

Safety evaluation of self-driving technologies has been extensively studied. One recent approach uses Monte Carlo based evaluation to estimate the occurrence probabilities of safety-critical events as safety measures. These Monte Carlo samples are generated from stochastic input models constructed based on real-world data. In this paper, we propose an approach to assess the impact on the probability estimates from the evaluation procedures due to the estimation error caused by data variability. Our proposed method merges the classical bootstrap method for estimating input uncertainty with a likelihood ratio based scheme to reuse experiment outputs. This approach is economical and efficient in terms of implementation costs in assessing input uncertainty for the evaluation of self-driving technology. We use an example in autonomous vehicle (AV) safety evaluation to demonstrate the proposed approach as a diagnostic tool for the quality of the fitted input model.