Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video

TL;DR

Restage4D is a novel pipeline that reanimates deformable 3D scenes from a single video, leveraging real-world motion priors to improve geometric consistency and correct generative artifacts in 4D scene synthesis.

Contribution

We introduce Restage4D, a geometry-preserving, video-conditioned 4D restaging method that uses a novel training strategy and loss functions to enhance deformable scene reanimation from minimal input.

Findings

Improved geometry consistency and motion quality on DAVIS and PointOdyssey datasets.

Automatic correction of generative model errors in 4D scene synthesis.

Enhanced 3D tracking performance in deformable scene reanimation.

Abstract

Creating deformable 3D content has gained increasing attention with the rise of text-to-image and image-to-video generative models. While these models provide rich semantic priors for appearance, they struggle to capture the physical realism and motion dynamics needed for authentic 4D scene synthesis. In contrast, real-world videos can provide physically grounded geometry and articulation cues that are difficult to hallucinate. One question is raised: \textit{Can we generate physically consistent 4D content by leveraging the motion priors of the real-world video}? In this work, we explore the task of reanimating deformable 3D scenes from a single video, using the original sequence as a supervisory signal to correct artifacts from synthetic motion. We introduce \textbf{Restage4D}, a geometry-preserving pipeline for video-conditioned 4D restaging. Our approach uses a video-rewinding training strategy to temporally bridge a real base video and a synthetic driving video via a shared motion representation. We further incorporate an occlusion-aware rigidity loss and a disocclusion backtracing mechanism to improve structural and geometry consistency under challenging motion. We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, revealing the potential of video prior in 4D restaging task. Source code and trained models will be released.