LLM-based Realistic Safety-Critical Driving Video Generation

TL;DR

This paper introduces a framework that uses Large Language Models to automatically generate safety-critical driving scenarios in CARLA, combined with realistic video synthesis for autonomous vehicle testing.

Contribution

It presents a novel LLM-based approach for automatic, controllable, and realistic safety-critical scenario generation in driving simulations.

Findings

Successfully generates diverse safety-critical scenarios.

Creates realistic driving videos from simulated scenes.

Facilitates testing of autonomous vehicles in rare edge cases.

Abstract

Designing diverse and safety-critical driving scenarios is essential for evaluating autonomous driving systems. In this paper, we propose a novel framework that leverages Large Language Models (LLMs) for few-shot code generation to automatically synthesize driving scenarios within the CARLA simulator, which has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics. Given a few example prompts and code samples, the LLM generates safety-critical scenario scripts that specify the behavior and placement of traffic participants, with a particular focus on collision events. To bridge the gap between simulation and real-world appearance, we integrate a video generation pipeline using Cosmos-Transfer1 with ControlNet, which converts rendered scenes into realistic driving videos. Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases, such as pedestrian crossings under occlusion or sudden vehicle cut-ins. Experimental results demonstrate the effectiveness of our method in generating a wide range of realistic, diverse, and safety-critical scenarios, offering a promising tool for simulation-based testing of autonomous vehicles.