Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles

TL;DR

This paper introduces ScenGE, a novel framework that generates diverse, plausible, and challenging safety-critical scenarios for autonomous vehicles using adversarial reasoning and complex traffic flow amplification, improving safety testing and robustness.

Contribution

We propose ScenGE, a framework combining large language models and scenario evolution to generate diverse, adversarial, and realistic safety-critical scenarios for autonomous vehicle testing.

Findings

ScenGE uncovers 31.96% more severe collision cases than baselines.

It applies to large models and various simulators, enhancing robustness.

Generated scenarios are validated as plausible and critical through real-world tests.

Abstract

The generation of safety-critical scenarios in simulation has become increasingly crucial for safety evaluation in autonomous vehicles prior to road deployment in society. However, current approaches largely rely on predefined threat patterns or rule-based strategies, which limit their ability to expose diverse and unforeseen failure modes. To overcome these, we propose ScenGE, a framework that can generate plentiful safety-critical scenarios by reasoning novel adversarial cases and then amplifying them with complex traffic flows. Given a simple prompt of a benign scene, it first performs Meta-Scenario Generation, where a large language model, grounded in structured driving knowledge, infers an adversarial agent whose behavior poses a threat that is both plausible and deliberately challenging. This meta-scenario is then specified in executable code for precise in-simulator control. Subsequently, Complex Scenario Evolution uses background vehicles to amplify the core threat introduced by Meta-Scenario. It builds an adversarial collaborator graph to identify key agent trajectories for optimization. These perturbations are designed to simultaneously reduce the ego vehicle's maneuvering space and create critical occlusions. Extensive experiments conducted on multiple reinforcement learning based AV models show that ScenGE uncovers more severe collision cases (+31.96%) on average than SoTA baselines. Additionally, our ScenGE can be applied to large model based AV systems and deployed on different simulators; we further observe that adversarial training on our scenarios improves the model robustness. Finally, we validate our framework through real-world vehicle tests and human evaluation, confirming that the generated scenarios are both plausible and critical. We hope our paper can build up a critical step towards building public trust and ensuring their safe deployment.