Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge

TL;DR

This paper introduces a novel framework combining CVAE and LLMs to generate realistic, risk-sensitive safety-critical scenarios for autonomous driving, enhancing safety validation by covering rare, challenging events.

Contribution

It presents a new high-fidelity scenario generation method that integrates a CVAE with an LLM for controllable, risk-aware simulation of complex traffic scenarios.

Findings

Increases coverage of high-risk events in simulations

Improves realism and distributional consistency with real-world data

Generates more challenging scenarios for autonomous system testing

Abstract

Autonomous driving faces critical challenges in rare long-tail events and complex multi-agent interactions, which are scarce in real-world data yet essential for robust safety validation. This paper presents a high-fidelity scenario generation framework that integrates a conditional variational autoencoder (CVAE) with a large language model (LLM). The CVAE encodes historical trajectories and map information from large-scale naturalistic datasets to learn latent traffic structures, enabling the generation of physically consistent base scenarios. Building on this, the LLM acts as an adversarial reasoning engine, parsing unstructured scene descriptions into domain-specific loss functions and dynamically guiding scenario generation across varying risk levels. This knowledge-driven optimization balances realism with controllability, ensuring that generated scenarios remain both plausible and risk-sensitive. Extensive experiments in CARLA and SMARTS demonstrate that our framework substantially increases the coverage of high-risk and long-tail events, improves consistency between simulated and real-world traffic distributions, and exposes autonomous driving systems to interactions that are significantly more challenging than those produced by existing rule- or data-driven methods. These results establish a new pathway for safety validation, enabling principled stress-testing of autonomous systems under rare but consequential events.