Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios

TL;DR

This paper presents a systematic, scenario-based procedure for assessing autonomous vehicle safety that balances thorough testing with confidentiality concerns, using limited physical tests and performance ratings.

Contribution

It introduces a novel approach to select and verify tests based on ODD and DDT, reducing physical testing needs while protecting proprietary information.

Findings

Test selection tailored to AV's ODD and DDT

Applicant provides performance ratings instead of detailed results

Limited physical tests suffice for verification

Abstract

The development of Autonomous Vehicles (AVs) has made significant progress in the last years. An important aspect in the development of AVs is the assessment of their safety. New approaches need to be worked out. Among these, real-world scenario-based assessment is widely supported by many players in the automotive field. Scenario-based assessment allows for using virtual simulation tools in addition to physical tests, such as on a test track, proving ground, or public road. We propose a procedure for real-world scenario-based road-approval assessment considering three stakeholders: the applicant, the assessor, and the (road or vehicle) authority. The challenges are as follows. Firstly, the tests need to be tailored to the operational design domain (ODD) and dynamic driving task (DDT) description of the AV. Secondly, it is assumed that the applicant does not want to disclose all of the detailed test results because of proprietary or confidential information contained in these results. Thirdly, due to the complex ODD and DDT, many test scenarios are required to obtain sufficient confidence in the assessment of the AV. Consequently, it is assumed that due to limited resources, it is infeasible for the assessor to conduct all (physical) tests. We propose a systematic approach for determining the tests that are based on the requirements set by the authority and the AV's ODD and DDT description, such that the tests are tailored to the applicable ODD and DDT. Each test comes with metrics that enables the applicant to provide a performance rating of the AV for each of the tests. By only providing a performance rating for each test, the applicant does not need to disclose the details of the test results. In our proposed procedure, the assessor only conducts a limited number of tests. The main purpose of these tests is to verify the fidelity of the results provided by the applicant.