GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model

TL;DR

GenDDS leverages advanced diffusion models and descriptive prompts to generate diverse, realistic driving videos, enhancing training datasets for autonomous vehicles and addressing the scarcity of rare scenario data.

Contribution

This work introduces a novel pipeline combining Stable Diffusion XL, ControlNet, and Hotshot-XL for high-quality, diverse driving scenario video generation from prompts.

Findings

Generated videos closely mimic real-world driving complexity.

The approach produces diverse scenarios including rare traffic conditions.

High-quality videos suitable for autonomous driving training.

Abstract

Autonomous driving training requires a diverse range of datasets encompassing various traffic conditions, weather scenarios, and road types. Traditional data augmentation methods often struggle to generate datasets that represent rare occurrences. To address this challenge, we propose GenDDS, a novel approach for generating driving scenarios generation by leveraging the capabilities of Stable Diffusion XL (SDXL), an advanced latent diffusion model. Our methodology involves the use of descriptive prompts to guide the synthesis process, aimed at producing realistic and diverse driving scenarios. With the power of the latest computer vision techniques, such as ControlNet and Hotshot-XL, we have built a complete pipeline for video generation together with SDXL. We employ the KITTI dataset, which includes real-world driving videos, to train the model. Through a series of experiments, we demonstrate that our model can generate high-quality driving videos that closely replicate the complexity and variability of real-world driving scenarios. This research contributes to the development of sophisticated training data for autonomous driving systems and opens new avenues for creating virtual environments for simulation and validation purposes.