Joint 3D Geometry Reconstruction and Motion Generation for 4D Synthesis from a Single Image

TL;DR

This paper introduces MoRe4D, a framework that jointly reconstructs 3D geometry and generates motion to synthesize dynamic 4D scenes from a single image, utilizing a new large-scale dataset and diffusion-based trajectory generation.

Contribution

It presents a novel joint reconstruction and motion generation framework for 4D scene synthesis from a single image, along with a new dataset and diffusion-based trajectory generator.

Findings

MoRe4D produces high-quality, multi-view consistent 4D scenes.

The method effectively integrates geometry and dynamics from a single image.

Experiments demonstrate rich dynamic details in synthesized 4D scenes.

Abstract

Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization. To address these, we extend the reconstruct-then-generate framework to jointly perform Motion generation and geometric Reconstruction for 4D Synthesis (MoRe4D). We first introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense point trajectories, addressing the scarcity of high-quality 4D scene data. Based on this, we propose a diffusion-based 4D Scene Trajectory Generator (4D-STraG) to jointly generate geometrically consistent and motion-plausible 4D point trajectories. To leverage single-view priors, we design a depth-guided motion normalization strategy and a motion-aware module for effective geometry and dynamics integration. We then propose a 4D View Synthesis Module (4D-ViSM) to render videos with arbitrary camera trajectories from 4D point track representations. Experiments show that MoRe4D generates high-quality 4D scenes with multi-view consistency and rich dynamic details from a single image. Code: https://github.com/Zhangyr2022/MoRe4D.