A Unified Approach for Text- and Image-guided 4D Scene Generation

TL;DR

Dream-in-4D introduces a novel two-stage diffusion-based method for generating high-quality, consistent 4D scenes from text prompts, effectively disentangling static assets and motion for flexible, controllable dynamic scene synthesis.

Contribution

It presents the first unified approach combining static 3D asset learning and motion modeling for text-to-4D scene generation using diffusion guidance.

Findings

Significantly improves image and motion quality in 4D generation.

Enhances 3D consistency and text fidelity over baseline methods.

Enables controllable 4D generation from images or text.

Abstract

Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images. However, the challenging problem of text-to-4D dynamic 3D scene generation with diffusion guidance remains largely unexplored. We propose Dream-in-4D, which features a novel two-stage approach for text-to-4D synthesis, leveraging (1) 3D and 2D diffusion guidance to effectively learn a high-quality static 3D asset in the first stage; (2) a deformable neural radiance field that explicitly disentangles the learned static asset from its deformation, preserving quality during motion learning; and (3) a multi-resolution feature grid for the deformation field with a displacement total variation loss to effectively learn motion with video diffusion guidance in the second stage. Through a user preference study, we demonstrate that our approach significantly advances image and motion quality, 3D consistency and text fidelity for text-to-4D generation compared to baseline approaches. Thanks to its motion-disentangled representation, Dream-in-4D can also be easily adapted for controllable generation where appearance is defined by one or multiple images, without the need to modify the motion learning stage. Thus, our method offers, for the first time, a unified approach for text-to-4D, image-to-4D and personalized 4D generation tasks.