From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios

TL;DR

This paper presents a novel approach combining Large Language Models with structured prompts to automatically evaluate and generate safety-critical driving scenarios, reducing reliance on manual scenario design and improving testing scalability for autonomous vehicles.

Contribution

The authors introduce LLM-guided evaluation and adversarial scenario generation techniques specifically tailored for autonomous vehicle safety testing, with new prompt strategies and an adversarial module.

Findings

Evaluation module accurately detects collision scenarios.

Generation module synthesizes realistic, high-risk scenarios.

Approach reduces dependence on handcrafted safety metrics.

Abstract

Ensuring the safety of autonomous vehicles requires virtual scenario-based testing, which depends on the robust evaluation and generation of safety-critical scenarios. So far, researchers have used scenario-based testing frameworks that rely heavily on handcrafted scenarios as safety metrics. To reduce the effort of human interpretation and overcome the limited scalability of these approaches, we combine Large Language Models (LLMs) with structured scenario parsing and prompt engineering to automatically evaluate and generate safety-critical driving scenarios. We introduce Cartesian and Ego-centric prompt strategies for scenario evaluation, and an adversarial generation module that modifies trajectories of risk-inducing vehicles (ego-attackers) to create critical scenarios. We validate our approach using a 2D simulation framework and multiple pre-trained LLMs. The results show that the evaluation module effectively detects collision scenarios and infers scenario safety. Meanwhile, the new generation module identifies high-risk agents and synthesizes realistic, safety-critical scenarios. We conclude that an LLM equipped with domain-informed prompting techniques can effectively evaluate and generate safety-critical driving scenarios, reducing dependence on handcrafted metrics. We release our open-source code and scenarios at: https://github.com/TUM-AVS/From-Words-to-Collisions.