UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass

TL;DR

UniSH introduces a novel feed-forward framework that unifies scene and human reconstruction at metric scale, effectively leveraging unlabeled in-the-wild data to improve generalization and fidelity in 3D modeling.

Contribution

The paper proposes a new training paradigm combining distillation and two-stage supervision to enhance 3D scene and human reconstruction from limited real-world data.

Findings

Achieves state-of-the-art results in human-centric scene reconstruction.

Outperforms existing methods in global human motion estimation.

Demonstrates strong generalization to in-the-wild videos.

Abstract

We present UniSH, a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction. A key challenge in this domain is the scarcity of large-scale, annotated real-world data, forcing a reliance on synthetic datasets. This reliance introduces a significant sim-to-real domain gap, leading to poor generalization, low-fidelity human geometry, and poor alignment on in-the-wild videos. To address this, we propose an innovative training paradigm that effectively leverages unlabeled in-the-wild data. Our framework bridges strong, disparate priors from scene reconstruction and HMR, and is trained with two core components: (1) a robust distillation strategy to refine human surface details by distilling high-frequency details from an expert depth model, and (2) a two-stage supervision scheme, which first learns coarse localization on synthetic data, then fine-tunes on real data by directly optimizing the geometric correspondence between the SMPL mesh and the human point cloud. This approach enables our feed-forward model to jointly recover high-fidelity scene geometry, human point clouds, camera parameters, and coherent, metric-scale SMPL bodies, all in a single forward pass. Extensive experiments demonstrate that our model achieves state-of-the-art performance on human-centric scene reconstruction and delivers highly competitive results on global human motion estimation, comparing favorably against both optimization-based frameworks and HMR-only methods. Project page: https://murphylmf.github.io/UniSH/