Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation

TL;DR

Dream4D introduces a novel framework that combines controllable video generation and neural 4D reconstruction to produce spatiotemporally consistent 4D content from limited input, addressing key challenges in view consistency and scene dynamics.

Contribution

It presents the first approach to integrate temporal priors from diffusion models with geometric reconstruction for 4D content generation from a single image.

Findings

Achieves higher quality metrics (mPSNR, mSSIM) than existing methods.

Successfully generates geometrically consistent multi-view sequences.

Demonstrates effective camera trajectory prediction from minimal input.

Abstract

The synthesis of spatiotemporally coherent 4D content presents fundamental challenges in computer vision, requiring simultaneous modeling of high-fidelity spatial representations and physically plausible temporal dynamics. Current approaches often struggle to maintain view consistency while handling complex scene dynamics, particularly in large-scale environments with multiple interacting elements. This work introduces Dream4D, a novel framework that bridges this gap through a synergy of controllable video generation and neural 4D reconstruction. Our approach seamlessly combines a two-stage architecture: it first predicts optimal camera trajectories from a single image using few-shot learning, then generates geometrically consistent multi-view sequences via a specialized pose-conditioned diffusion process, which are finally converted into a persistent 4D representation. This framework is the first to leverage both rich temporal priors from video diffusion models and geometric awareness of the reconstruction models, which significantly facilitates 4D generation and shows higher quality (e.g., mPSNR, mSSIM) over existing methods.