CoCo4D: Comprehensive and Complex 4D Scene Generation

TL;DR

CoCo4D is a novel framework that generates detailed, multi-view consistent 4D scenes from text prompts by modeling foreground motion and background evolution separately, ensuring realistic and immersive dynamic scene synthesis.

Contribution

The paper introduces CoCo4D, a new method that divides 4D scene generation into foreground and background modeling guided by motion, enabling more realistic and coherent dynamic scene synthesis from text and images.

Findings

Achieves comparable or superior 4D scene generation quality to existing methods.

Effectively models articulated foreground motion and background changes.

Demonstrates efficiency and realism in dynamic scene synthesis.

Abstract

Existing 4D synthesis methods primarily focus on object-level generation or dynamic scene synthesis with limited novel views, restricting their ability to generate multi-view consistent and immersive dynamic 4D scenes. To address these constraints, we propose a framework (dubbed as CoCo4D) for generating detailed dynamic 4D scenes from text prompts, with the option to include images. Our method leverages the crucial observation that articulated motion typically characterizes foreground objects, whereas background alterations are less pronounced. Consequently, CoCo4D divides 4D scene synthesis into two responsibilities: modeling the dynamic foreground and creating the evolving background, both directed by a reference motion sequence. Given a text prompt and an optional reference image, CoCo4D first generates an initial motion sequence utilizing video diffusion models. This motion sequence then guides the synthesis of both the dynamic foreground object and the background using a novel progressive outpainting scheme. To ensure seamless integration of the moving foreground object within the dynamic background, CoCo4D optimizes a parametric trajectory for the foreground, resulting in realistic and coherent blending. Extensive experiments show that CoCo4D achieves comparable or superior performance in 4D scene generation compared to existing methods, demonstrating its effectiveness and efficiency. More results are presented on our website https://colezwhy.github.io/coco4d/.