DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction

TL;DR

DP-DeGauss is a novel probabilistic framework for egocentric 4D scene reconstruction that effectively disentangles background, hands, and objects, improving dynamic scene modeling and editing capabilities.

Contribution

It introduces a dynamic Gaussian decomposition approach with category-specific masks and motion control, enabling explicit separation of scene components in egocentric videos.

Findings

Outperforms baselines with +1.70dB PSNR gain

Achieves state-of-the-art disentanglement of scene components

Enables fine-grained scene understanding and editing

Abstract

Egocentric video is crucial for next-generation 4D scene reconstruction, with applications in AR/VR and embodied AI. However, reconstructing dynamic first-person scenes is challenging due to complex ego-motion, occlusions, and hand-object interactions. Existing decomposition methods are ill-suited, assuming fixed viewpoints or merging dynamics into a single foreground. To address these limitations, we introduce DP-DeGauss, a dynamic probabilistic Gaussian decomposition framework for egocentric 4D reconstruction. Our method initializes a unified 3D Gaussian set from COLMAP priors, augments each with a learnable category probability, and dynamically routes them into specialized deformation branches for background, hands, or object modeling. We employ category-specific masks for better disentanglement and introduce brightness and motion-flow control to improve static rendering and dynamic reconstruction. Extensive experiments show that DP-DeGauss outperforms baselines by +1.70dB in PSNR on average with SSIM and LPIPS gains. More importantly, our framework achieves the first and state-of-the-art disentanglement of background, hand, and object components, enabling explicit, fine-grained separation, paving the way for more intuitive ego scene understanding and editing.