Hand3R: Online 4D Hand-Scene Reconstruction in the Wild

TL;DR

Hand3R is an innovative online framework that jointly reconstructs dynamic hands and dense 3D scene geometry from monocular video, advancing Embodied AI understanding of physical interactions.

Contribution

It introduces the first online method for simultaneous 4D hand and scene reconstruction using a scene-aware prompting mechanism and high-fidelity hand priors.

Findings

Achieves accurate hand mesh reconstruction in real-time

Reconstructs dense scene geometry with metric scale

Operates without offline optimization

Abstract

For Embodied AI, jointly reconstructing dynamic hands and the dense scene context is crucial for understanding physical interaction. However, most existing methods recover isolated hands in local coordinates, overlooking the surrounding 3D environment. To address this, we present Hand3R, the first online framework for joint 4D hand-scene reconstruction from monocular video. Hand3R synergizes a pre-trained hand expert with a 4D scene foundation model via a scene-aware visual prompting mechanism. By injecting high-fidelity hand priors into a persistent scene memory, our approach enables simultaneous reconstruction of accurate hand meshes and dense metric-scale scene geometry in a single forward pass. Experiments demonstrate that Hand3R bypasses the reliance on offline optimization and delivers competitive performance in both local hand reconstruction and global positioning.