SirenPose: Dynamic Scene Reconstruction via Geometric Supervision

TL;DR

SirenPose is a novel method for dynamic scene reconstruction from monocular videos that combines geometric supervision, physics-inspired constraints, and high-frequency signal modeling to achieve accurate, consistent, and detailed 3D reconstructions.

Contribution

It introduces a geometry-aware loss with sinusoidal networks, expands the UniKPT dataset, and employs graph neural networks for improved dynamic scene and pose estimation.

Findings

Outperforms state-of-the-art on Sintel, Bonn, and DAVIS benchmarks.

Reduces FVD by 17.8%, FID by 28.7%, and improves LPIPS by 6%.

Enhances temporal consistency and geometric accuracy in dynamic scenes.

Abstract

We introduce SirenPose, a geometry-aware loss formulation that integrates the periodic activation properties of sinusoidal representation networks with keypoint-based geometric supervision, enabling accurate and temporally consistent reconstruction of dynamic 3D scenes from monocular videos. Existing approaches often struggle with motion fidelity and spatiotemporal coherence in challenging settings involving fast motion, multi-object interaction, occlusion, and rapid scene changes. SirenPose incorporates physics inspired constraints to enforce coherent keypoint predictions across both spatial and temporal dimensions, while leveraging high frequency signal modeling to capture fine grained geometric details. We further expand the UniKPT dataset to 600,000 annotated instances and integrate graph neural networks to model keypoint relationships and structural correlations. Extensive experiments on benchmarks including Sintel, Bonn, and DAVIS demonstrate that SirenPose consistently outperforms state-of-the-art methods. On DAVIS, SirenPose achieves a 17.8 percent reduction in FVD, a 28.7 percent reduction in FID, and a 6.0 percent improvement in LPIPS compared to MoSCA. It also improves temporal consistency, geometric accuracy, user score, and motion smoothness. In pose estimation, SirenPose outperforms Monst3R with lower absolute trajectory error as well as reduced translational and rotational relative pose error, highlighting its effectiveness in handling rapid motion, complex dynamics, and physically plausible reconstruction.