Controlling Space and Time with Diffusion Models

TL;DR

This paper introduces 4DiM, a novel diffusion-based model for 4D view synthesis that supports arbitrary camera trajectories and timestamps, improving generalization and enabling dynamic scene generation from limited data.

Contribution

The paper presents the first NVS method with intuitive metric-scale camera control and a new architecture that trains on mixed 3D, 4D, and video data for better generalization.

Findings

Outperforms prior 3D NVS models in fidelity and pose accuracy

Enables scene dynamics and pose-controlled video generation

Supports various tasks like single-image 3D and video translation

Abstract

We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS), supporting generation with arbitrary camera trajectories and timestamps, in natural scenes, conditioned on one or more images. With a novel architecture and sampling procedure, we enable training on a mixture of 3D (with camera pose), 4D (pose+time) and video (time but no pose) data, which greatly improves generalization to unseen images and camera pose trajectories over prior works that focus on limited domains (e.g., object centric). 4DiM is the first-ever NVS method with intuitive metric-scale camera pose control enabled by our novel calibration pipeline for structure-from-motion-posed data. Experiments demonstrate that 4DiM outperforms prior 3D NVS models both in terms of image fidelity and pose alignment, while also enabling the generation of scene dynamics. 4DiM provides a general framework for a variety of tasks including single-image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation, which we illustrate qualitatively on a variety of scenes. For an overview see https://4d-diffusion.github.io