Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention

TL;DR

CogDriving is a novel diffusion transformer-based network that generates high-quality multi-view driving videos by capturing holistic spatio-temporal-view associations, with a lightweight controller and dynamic object learning for improved realism.

Contribution

The paper introduces CogDriving, a diffusion transformer architecture with holistic 4D attention and a lightweight controller, advancing multi-view driving video synthesis with better consistency and control.

Findings

Achieves an FVD score of 37.8 on nuScenes validation set.

Outperforms existing methods in multi-view video generation quality.

Demonstrates effective control over Bird's-Eye-View layouts.

Abstract

Generating multi-view videos for autonomous driving training has recently gained much attention, with the challenge of addressing both cross-view and cross-frame consistency. Existing methods typically apply decoupled attention mechanisms for spatial, temporal, and view dimensions. However, these approaches often struggle to maintain consistency across dimensions, particularly when handling fast-moving objects that appear at different times and viewpoints. In this paper, we present CogDriving, a novel network designed for synthesizing high-quality multi-view driving videos. CogDriving leverages a Diffusion Transformer architecture with holistic-4D attention modules, enabling simultaneous associations across the spatial, temporal, and viewpoint dimensions. We also propose a lightweight controller tailored for CogDriving, i.e., Micro-Controller, which uses only 1.1% of the parameters of the standard ControlNet, enabling precise control over Bird's-Eye-View layouts. To enhance the generation of object instances crucial for autonomous driving, we propose a re-weighted learning objective, dynamically adjusting the learning weights for object instances during training. CogDriving demonstrates strong performance on the nuScenes validation set, achieving an FVD score of 37.8, highlighting its ability to generate realistic driving videos. The project can be found at https://luhannan.github.io/CogDrivingPage/.