4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency

TL;DR

4DGen introduces a controllable 4D content creation framework using monocular videos and dynamic 3D Gaussians, enabling high-quality, realistic, and grounded 4D content with improved motion control and consistency.

Contribution

The paper presents a novel 4D content generation pipeline that leverages monocular videos and dynamic 3D Gaussians for controllable, high-resolution, and consistent 4D synthesis, surpassing prior methods in quality and control.

Findings

Outperforms existing video-to-4D methods in reconstruction fidelity.

Enables user-controlled motion via monocular videos or image-to-video generation.

Produces realistic, high-quality 4D content with spatial-temporal consistency.

Abstract

Aided by text-to-image and text-to-video diffusion models, existing 4D content creation pipelines utilize score distillation sampling to optimize the entire dynamic 3D scene. However, as these pipelines generate 4D content from text or image inputs directly, they are constrained by limited motion capabilities and depend on unreliable prompt engineering for desired results. To address these problems, this work introduces \textbf{4DGen}, a novel framework for grounded 4D content creation. We identify monocular video sequences as a key component in constructing the 4D content. Our pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations, thus offering superior control over content creation. Furthermore, we construct our 4D representation using dynamic 3D Gaussians, which permits efficient, high-resolution supervision through rendering during training, thereby facilitating high-quality 4D generation. Additionally, we employ spatial-temporal pseudo labels on anchor frames, along with seamless consistency priors implemented through 3D-aware score distillation sampling and smoothness regularizations. Compared to existing video-to-4D baselines, our approach yields superior results in faithfully reconstructing input signals and realistically inferring renderings from novel viewpoints and timesteps. More importantly, compared to previous image-to-4D and text-to-4D works, 4DGen supports grounded generation, offering users enhanced control and improved motion generation capabilities, a feature difficult to achieve with previous methods. Project page: https://vita-group.github.io/4DGen/