UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving

TL;DR

UniMLVG is a comprehensive framework that generates long, multi-view driving videos with high quality and precise control, significantly advancing autonomous driving simulation capabilities.

Contribution

It introduces a unified diffusion-based model with explicit viewpoint control and multi-view training, improving diversity, quality, and motion consistency in long multi-view videos.

Findings

Achieves 48.2% improvement in FID over comparable models.

Achieves 35.2% improvement in FVD.

Effectively handles various input formats and control constraints.

Abstract

The creation of diverse and realistic driving scenarios has become essential to enhance perception and planning capabilities of the autonomous driving system. However, generating long-duration, surround-view consistent driving videos remains a significant challenge. To address this, we present UniMLVG, a unified framework designed to generate extended street multi-perspective videos under precise control. By integrating single- and multi-view driving videos into the training data, our approach updates a DiT-based diffusion model equipped with cross-frame and cross-view modules across three stages with multi training objectives, substantially boosting the diversity and quality of generated visual content. Importantly, we propose an innovative explicit viewpoint modeling approach for multi-view video generation to effectively improve motion transition consistency. Capable of handling various input reference formats (e.g., text, images, or video), our UniMLVG generates high-quality multi-view videos according to the corresponding condition constraints such as 3D bounding boxes or frame-level text descriptions. Compared to the best models with similar capabilities, our framework achieves improvements of 48.2% in FID and 35.2% in FVD.