Synergistic Global-space Camera and Human Reconstruction from Videos

TL;DR

This paper presents SynCHMR, a novel approach that jointly reconstructs camera trajectories, human meshes, and scene point clouds from videos by integrating SLAM and human mesh recovery techniques.

Contribution

It introduces Human-aware Metric SLAM and a Scene-aware SMPL Denoiser to achieve consistent, metric-scale reconstructions of scenes and humans from monocular videos.

Findings

Achieves metric-scale camera and scene reconstruction from monocular videos.

Enhances human mesh recovery with scene and temporal coherence.

Provides a unified framework for scene and human reconstruction.

Abstract

Remarkable strides have been made in reconstructing static scenes or human bodies from monocular videos. Yet, the two problems have largely been approached independently, without much synergy. Most visual SLAM methods can only reconstruct camera trajectories and scene structures up to scale, while most HMR methods reconstruct human meshes in metric scale but fall short in reasoning with cameras and scenes. This work introduces Synergistic Camera and Human Reconstruction (SynCHMR) to marry the best of both worlds. Specifically, we design Human-aware Metric SLAM to reconstruct metric-scale camera poses and scene point clouds using camera-frame HMR as a strong prior, addressing depth, scale, and dynamic ambiguities. Conditioning on the dense scene recovered, we further learn a Scene-aware SMPL Denoiser to enhance world-frame HMR by incorporating spatio-temporal coherency and dynamic scene constraints. Together, they lead to consistent reconstructions of camera trajectories, human meshes, and dense scene point clouds in a common world frame. Project page: https://paulchhuang.github.io/synchmr