MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips

TL;DR

MagicHOI introduces a novel approach that uses large-scale view synthesis diffusion models as priors to improve 3D hand-object reconstruction from short monocular videos, especially when object visibility is limited.

Contribution

It integrates view synthesis diffusion priors into hand-object reconstruction, enabling accurate results without full object visibility or extensive paired data.

Findings

Outperforms existing state-of-the-art methods

Effectively regularizes unseen object regions

Enhances 3D reconstruction accuracy

Abstract

Most RGB-based hand-object reconstruction methods rely on object templates, while template-free methods typically assume full object visibility. This assumption often breaks in real-world settings, where fixed camera viewpoints and static grips leave parts of the object unobserved, resulting in implausible reconstructions. To overcome this, we present MagicHOI, a method for reconstructing hands and objects from short monocular interaction videos, even under limited viewpoint variation. Our key insight is that, despite the scarcity of paired 3D hand-object data, large-scale novel view synthesis diffusion models offer rich object supervision. This supervision serves as a prior to regularize unseen object regions during hand interactions. Leveraging this insight, we integrate a novel view synthesis model into our hand-object reconstruction framework. We further align hand to object by incorporating visible contact constraints. Our results demonstrate that MagicHOI significantly outperforms existing state-of-the-art hand-object reconstruction methods. We also show that novel view synthesis diffusion priors effectively regularize unseen object regions, enhancing 3D hand-object reconstruction.