RCG: Safety-Critical Scenario Generation for Robust Autonomous Driving via Real-World Crash Grounding

TL;DR

This paper introduces RCG, a framework that generates realistic, safety-critical driving scenarios by grounding adversarial perturbations in crash-informed semantics, improving autonomous vehicle testing and training.

Contribution

We develop a crash-informed semantic embedding and integrate it into scenario generation pipelines, enhancing the realism and effectiveness of safety-critical scenario creation for autonomous driving.

Findings

Generated scenarios lead to 9.2% higher success rates in downstream tasks.

The approach produces more plausible adversary behaviors.

Enhanced stress testing of autonomous systems.

Abstract

Safety-critical scenarios are essential for training and evaluating autonomous driving (AD) systems, yet remain extremely rare in real-world driving datasets. To address this, we propose Real-world Crash Grounding (RCG), a scenario generation framework that integrates crash-informed semantics into adversarial perturbation pipelines. We construct a safety-aware behavior representation through contrastive pre-training on large-scale driving logs, followed by fine-tuning on a small, crash-rich dataset with approximate trajectory annotations extracted from video. This embedding captures semantic structure aligned with real-world accident behaviors and supports selection of adversary trajectories that are both high-risk and behaviorally realistic. We incorporate the resulting selection mechanism into two prior scenario generation pipelines, replacing their handcrafted scoring objectives with an embedding-based criterion. Experimental results show that ego agents trained against these generated scenarios achieve consistently higher downstream success rates, with an average improvement of 9.2% across seven evaluation settings. Qualitative and quantitative analyses further demonstrate that our approach produces more plausible and nuanced adversary behaviors, enabling more effective and realistic stress testing of AD systems. Code and tools will be released publicly.