World-consistent Video Diffusion with Explicit 3D Modeling

TL;DR

This paper introduces WVD, a diffusion-based framework that explicitly models 3D consistency in video generation by learning joint RGB and XYZ representations, enabling flexible multi-view and camera-controlled synthesis.

Contribution

The paper presents a novel diffusion transformer that incorporates explicit 3D supervision with XYZ images, unifying multiple tasks like 3D generation, multi-view stereo, and camera-driven video synthesis.

Findings

Competitive performance on multiple benchmarks

Supports multi-task adaptability with inpainting strategy

Enables 3D-consistent video and image generation

Abstract

Recent advancements in diffusion models have set new benchmarks in image and video generation, enabling realistic visual synthesis across single- and multi-frame contexts. However, these models still struggle with efficiently and explicitly generating 3D-consistent content. To address this, we propose World-consistent Video Diffusion (WVD), a novel framework that incorporates explicit 3D supervision using XYZ images, which encode global 3D coordinates for each image pixel. More specifically, we train a diffusion transformer to learn the joint distribution of RGB and XYZ frames. This approach supports multi-task adaptability via a flexible inpainting strategy. For example, WVD can estimate XYZ frames from ground-truth RGB or generate novel RGB frames using XYZ projections along a specified camera trajectory. In doing so, WVD unifies tasks like single-image-to-3D generation, multi-view stereo, and camera-controlled video generation. Our approach demonstrates competitive performance across multiple benchmarks, providing a scalable solution for 3D-consistent video and image generation with a single pretrained model.