DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation

TL;DR

DriveDreamer-2 introduces a novel LLM-enhanced world model for generating diverse, customized driving videos that improve autonomous driving training and outperform existing methods in quality.

Contribution

It is the first to integrate LLMs with world models for user-defined, multi-view driving video generation, enhancing customization and video quality.

Findings

Generates diverse, user-defined driving videos with high realism.

Improves training for perception tasks like 3D detection and tracking.

Achieves superior FID and FVD scores compared to state-of-the-art methods.

Abstract

World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.