Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion

TL;DR

This paper introduces Vid3D, a novel approach that generates dynamic 3D scenes from 2D video diffusion models by independently creating 3D representations for each timestep, challenging the necessity of explicit temporal modeling.

Contribution

Vid3D demonstrates that high-quality 3D video generation can be achieved without explicitly modeling 3D temporal dynamics, simplifying the process compared to prior methods.

Findings

Vid3D achieves comparable results to state-of-the-art methods.

Performance remains robust with fewer views per frame.

Explicit 3D temporal modeling may not be essential for high-quality 3D video synthesis.

Abstract

A recent frontier in computer vision has been the task of 3D video generation, which consists of generating a time-varying 3D representation of a scene. To generate dynamic 3D scenes, current methods explicitly model 3D temporal dynamics by jointly optimizing for consistency across both time and views of the scene. In this paper, we instead investigate whether it is necessary to explicitly enforce multiview consistency over time, as current approaches do, or if it is sufficient for a model to generate 3D representations of each timestep independently. We hence propose a model, Vid3D, that leverages 2D video diffusion to generate 3D videos by first generating a 2D "seed" of the video's temporal dynamics and then independently generating a 3D representation for each timestep in the seed video. We evaluate Vid3D against two state-of-the-art 3D video generation methods and find that Vid3D is achieves comparable results despite not explicitly modeling 3D temporal dynamics. We further ablate how the quality of Vid3D depends on the number of views generated per frame. While we observe some degradation with fewer views, performance degradation remains minor. Our results thus suggest that 3D temporal knowledge may not be necessary to generate high-quality dynamic 3D scenes, potentially enabling simpler generative algorithms for this task.