UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Driving

TL;DR

UniDriveDreamer introduces a unified multimodal world model for autonomous driving that directly generates synchronized multi-camera and LiDAR observations, improving data synthesis and downstream task performance.

Contribution

It presents a novel single-stage framework with modality-specific VAEs, a cross-modal alignment technique, and a diffusion transformer for joint multimodal future observation generation.

Findings

Outperforms previous methods in multimodal data synthesis

Improves downstream autonomous driving tasks

Demonstrates stable training and cross-modal consistency

Abstract

World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream