From Dashcam Videos to Driving Simulations: Stress Testing Automated Vehicles against Rare Events

TL;DR

This paper presents an automated framework that converts real-world dashcam videos into detailed simulation scenarios for testing Automated Driving Systems, significantly reducing manual effort and time while maintaining high fidelity.

Contribution

The authors introduce a novel, fully automated method using Video Language Models to transform real crash videos into simulation scenarios with adjustable parameters for flexible testing.

Findings

Conversion time reduced to minutes with full automation

High fidelity in capturing original driving behaviors

Framework enables flexible scenario parameter adjustments

Abstract

Testing Automated Driving Systems (ADS) in simulation with realistic driving scenarios is important for verifying their performance. However, converting real-world driving videos into simulation scenarios is a significant challenge due to the complexity of interpreting high-dimensional video data and the time-consuming nature of precise manual scenario reconstruction. In this work, we propose a novel framework that automates the conversion of real-world car crash videos into detailed simulation scenarios for ADS testing. Our approach leverages prompt-engineered Video Language Models(VLM) to transform dashcam footage into SCENIC scripts, which define the environment and driving behaviors in the CARLA simulator, enabling the generation of realistic simulation scenarios. Importantly, rather than solely aiming for one-to-one scenario reconstruction, our framework focuses on capturing the essential driving behaviors from the original video while offering flexibility in parameters such as weather or road conditions to facilitate search-based testing. Additionally, we introduce a similarity metric that helps iteratively refine the generated scenario through feedback by comparing key features of driving behaviors between the real and simulated videos. Our preliminary results demonstrate substantial time efficiency, finishing the real-to-sim conversion in minutes with full automation and no human intervention, while maintaining high fidelity to the original driving events.