On-Demand Scenario Generation for Testing Automated Driving Systems

TL;DR

This paper introduces the On-Demand Scenario Generation framework that creates diverse, risk-controlled driving scenarios for testing automated driving systems, improving safety evaluation by systematically varying scenario risk levels.

Contribution

The paper presents a novel framework that learns from real traffic data and uses risk regulation and heuristic search to generate diverse scenarios with controllable risk levels for ADS testing.

Findings

OSG effectively generates scenarios with varying risk levels.

It enables systematic comparison of ADS performance across risk levels.

Demonstrates importance of risk-controlled scenario generation in safety testing.

Abstract

The safety and reliability of Automated Driving Systems (ADS) are paramount, necessitating rigorous testing methodologies to uncover potential failures before deployment. Traditional testing approaches often prioritize either natural scenario sampling or safety-critical scenario generation, resulting in overly simplistic or unrealistic hazardous tests. In practice, the demand for natural scenarios (e.g., when evaluating the ADS's reliability in real-world conditions), critical scenarios (e.g., when evaluating safety in critical situations), or somewhere in between (e.g., when testing the ADS in regions with less civilized drivers) varies depending on the testing objectives. To address this issue, we propose the On-demand Scenario Generation (OSG) Framework, which generates diverse scenarios with varying risk levels. Achieving the goal of OSG is challenging due to the complexity of quantifying the criticalness and naturalness stemming from intricate vehicle-environment interactions, as well as the need to maintain scenario diversity across various risk levels. OSG learns from real-world traffic datasets and employs a Risk Intensity Regulator to quantitatively control the risk level. It also leverages an improved heuristic search method to ensure scenario diversity. We evaluate OSG on the Carla simulators using various ADSs. We verify OSG's ability to generate scenarios with different risk levels and demonstrate its necessity by comparing accident types across risk levels. With the help of OSG, we are now able to systematically and objectively compare the performance of different ADSs based on different risk levels.