Coverage Metrics for a Scenario Database for the Scenario-Based Assessment of Automated Driving Systems

TL;DR

This paper introduces quantitative coverage metrics to evaluate how well collected driving scenarios represent an ADS's operational domain and the driving data, ensuring comprehensive safety assessment.

Contribution

It proposes novel coverage metrics for assessing the completeness of scenario datasets in relation to the ADS's operational design domain and driving data.

Findings

100% coverage achievable under certain conditions

Metrics identify data gaps for improved coverage

Experiment with 200,000 scenarios from the HighD dataset

Abstract

Automated Driving Systems (ADSs) have the potential to make mobility services available and safe for all. A multi-pillar Safety Assessment Framework (SAF) has been proposed for the type-approval process of ADSs. The SAF requires that the test scenarios for the ADS adequately covers the Operational Design Domain (ODD) of the ADS. A common method for generating test scenarios involves basing them on scenarios identified and characterized from driving data. This work addresses two questions when collecting scenarios from driving data. First, do the collected scenarios cover all relevant aspects of the ADS' ODD? Second, do the collected scenarios cover all relevant aspects that are in the driving data, such that no potentially important situations are missed? This work proposes coverage metrics that provide a quantitative answer to these questions. The proposed coverage metrics are illustrated by means of an experiment in which over 200000 scenarios from 10 different scenario categories are collected from the HighD data set. The experiment demonstrates that a coverage of 100 % can be achieved under certain conditions, and it also identifies which data and scenarios could be added to enhance the coverage outcomes in case a 100 % coverage has not been achieved. Whereas this work presents metrics for the quantification of the coverage of driving data and the identified scenarios, this paper concludes with future research directions, including the quantification of the completeness of driving data and the identified scenarios.