View-Consistent Diffusion Representations for 3D-Consistent Video Generation

TL;DR

This paper introduces ViCoDR, a novel method that enhances 3D consistency in diffusion-based video generation, reducing artifacts and improving multi-view coherence across various generation tasks.

Contribution

The paper proposes ViCoDR, a new approach for learning multi-view consistent diffusion representations, significantly improving 3D consistency in generated videos.

Findings

ViCoDR improves 3D consistency in video generation.

Enhanced multi-view coherence reduces visual artifacts.

Method performs well across image-to-video, text-to-video, and multi-view tasks.

Abstract

Video generation models have made significant progress in generating realistic content, enabling applications in simulation, gaming, and film making. However, current generated videos still contain visual artifacts arising from 3D inconsistencies, e.g., objects and structures deforming under changes in camera pose, which can undermine user experience and simulation fidelity. Motivated by recent findings on representation alignment for diffusion models, we hypothesize that improving the multi-view consistency of video diffusion representations will yield more 3D-consistent video generation. Through detailed analysis on multiple recent camera-controlled video diffusion models we reveal strong correlations between 3D-consistent representations and videos. We also propose ViCoDR, a new approach for improving the 3D consistency of video models by learning multi-view consistent diffusion representations. We evaluate ViCoDR on camera controlled image-to-video, text-to-video, and multi-view generation models, demonstrating significant improvements in the 3D consistency of the generated videos. Project page: https://danier97.github.io/ViCoDR.