SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data

TL;DR

SAFE is a framework that uses advanced language model techniques to accurately extract and reconstruct realistic, scenario-driven crash scenarios from multimodal crash data, significantly improving autonomous driving system testing.

Contribution

The paper introduces SAFE, a novel multimodal crash data extraction framework that leverages retrieval-augmented generation, knowledge-grounded prompting, and self-validation to enhance scenario reconstruction accuracy.

Findings

Achieves 93.8% accuracy in road network extraction.

Outperforms existing methods in safety violation detection.

Reproduces more real-world crash cases with statistical significance.

Abstract

Ensuring the safety of Autonomous Driving Systems (ADS) requires realistic and reproducible test scenarios, yet extracting such scenarios from multimodal crash reports remains a major challenge. Large Language Models (LLMs) often hallucinate and lose map structure, resulting in unrealistic road layouts and vehicle behaviors. To address this, we introduce SAFE, a novel Scenario-based ADS testing Framework via multimodal Extraction, which leverages Retrieval-Augmented Generation (RAG), knowledge-grounded prompting, Chain-of-Thought (CoT) reasoning, and self-validation to improve scenario reconstruction from multimodal crash data. SAFE achieves 93.8% accuracy in extracting road network details, 80.0% for actor information, and 100% for environmental context. In human studies, SAFE outperforms LCTGen and AC3R in reconstructing consistent road networks and vehicle behaviors. Under identical ADS and simulator settings, SAFE detects 39 and 71 more safety violations than LCTGen and AC3R, respectively, and reproduces 12 more real-world crash cases than LCTGen. On 19 cases supported by AC3R, SAFE reproduces one additional crash case with statistically significant gains across five runs. It generates scenarios within 25 seconds and triggers violations after just 1 case (IDM) and 3 cases (PPO) in MetaDrive, as well as 1 case (Auto) in BeamNG. Code: https://github.com/Siwei-Luo-MQ/SAFE-ADS-Testing