Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features

TL;DR

This paper introduces a systematic method using Instance Space Analysis and machine learning to identify key features that determine the safety-criticality of autonomous vehicle test scenarios, improving testing efficiency.

Contribution

The paper presents a novel approach combining Instance Space Analysis with machine learning to predict safety outcomes of AV scenarios without extensive simulation.

Findings

High accuracy in classifying safe and unsafe scenarios

Effective visualization of feature impact on safety outcomes

Identification of untested regions in scenario feature space

Abstract

Ensuring the safety of autonomous vehicles (AVs) is of utmost importance and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road's structural features (lanes, turns, road speed, etc.). In this paper, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2D. This visualization helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritization.