DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos

TL;DR

DGS-LRM is a real-time feed-forward model that reconstructs deformable 3D scenes from monocular videos, enabling dynamic scene understanding and long-range 3D tracking with high accuracy.

Contribution

The paper introduces a novel deformable 3D Gaussian representation, an enhanced synthetic dataset, and a transformer-based model for real-time dynamic scene reconstruction from monocular videos.

Findings

Achieves reconstruction quality comparable to optimization-based methods.

Outperforms existing predictive methods on real-world dynamic scenes.

Enables accurate long-range 3D tracking with physically grounded deformation.

Abstract

We introduce the Deformable Gaussian Splats Large Reconstruction Model (DGS-LRM), the first feed-forward method predicting deformable 3D Gaussian splats from a monocular posed video of any dynamic scene. Feed-forward scene reconstruction has gained significant attention for its ability to rapidly create digital replicas of real-world environments. However, most existing models are limited to static scenes and fail to reconstruct the motion of moving objects. Developing a feed-forward model for dynamic scene reconstruction poses significant challenges, including the scarcity of training data and the need for appropriate 3D representations and training paradigms. To address these challenges, we introduce several key technical contributions: an enhanced large-scale synthetic dataset with ground-truth multi-view videos and dense 3D scene flow supervision; a per-pixel deformable 3D Gaussian representation that is easy to learn, supports high-quality dynamic view synthesis, and enables long-range 3D tracking; and a large transformer network that achieves real-time, generalizable dynamic scene reconstruction. Extensive qualitative and quantitative experiments demonstrate that DGS-LRM achieves dynamic scene reconstruction quality comparable to optimization-based methods, while significantly outperforming the state-of-the-art predictive dynamic reconstruction method on real-world examples. Its predicted physically grounded 3D deformation is accurate and can readily adapt for long-range 3D tracking tasks, achieving performance on par with state-of-the-art monocular video 3D tracking methods.