VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion

TL;DR

This paper introduces a hierarchical framework combining Vision Language Models and guided diffusion to generate realistic, diverse, and interactive safety-critical testing scenarios for autonomous driving systems, addressing real-time response challenges.

Contribution

It presents a novel hierarchical architecture integrating VLMs with adaptive guided diffusion for real-time, high-fidelity scenario generation in autonomous vehicle testing.

Findings

Efficient generation of realistic safety-critical scenarios.

High diversity and interactivity in generated scenarios.

Validated adaptability and VLM-guided performance through case studies.

Abstract

The safe deployment of autonomous driving systems (ADSs) relies on comprehensive testing and evaluation. However, safety-critical scenarios that can effectively expose system vulnerabilities are extremely sparse in the real world. Existing scenario generation methods face challenges in efficiently constructing long-tail scenarios that ensure fidelity, criticality, and interactivity, while particularly lacking real-time dynamic response capabilities to the vehicle under test (VUT). To address these challenges, this paper proposes a safety-critical testing scenario generation framework that integrates the high-level semantic understanding capabilities of Vision Language Models (VLMs) with the fine-grained generation capabilities of adaptive guided diffusion models. The framework establishes a three-layer hierarchical architecture comprising a strategic layer for VLM-directed scenario generation objective determination, a tactical layer for guidance function formulation, and an operational layer for guided diffusion execution. We first establish a high-quality fundamental diffusion model that learns the data distribution of real driving scenarios. Next, we design an adaptive guided diffusion method that enables real-time, precise control of background vehicles (BVs) in closed-loop simulation. The VLM is then incorporated to autonomously generate scenario generation objectives and guidance functions through deep scenario understanding and risk reasoning, ultimately guiding the diffusion model to achieve VLM-directed scenario generation. Experimental results demonstrate that the proposed method can efficiently generate realistic, diverse, and highly interactive safety-critical testing scenarios. Furthermore, case studies validate the adaptability and VLM-directed generation performance of the proposed method.