Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera

TL;DR

Dyn-HaMR is a novel method that reconstructs 4D global hand motion from monocular videos captured by dynamic cameras, addressing limitations of previous approaches and improving accuracy in real-world scenarios.

Contribution

It introduces a multi-stage optimization pipeline combining SLAM, hand priors, and hierarchical initialization for accurate 4D hand motion recovery from moving camera videos.

Findings

Outperforms existing methods in 4D hand mesh reconstruction

Works effectively on in-the-wild and indoor datasets

Establishes a new benchmark for monocular hand motion recovery

Abstract

We propose Dyn-HaMR, to the best of our knowledge, the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild. Reconstructing accurate 3D hand meshes from monocular videos is a crucial task for understanding human behaviour, with significant applications in augmented and virtual reality (AR/VR). However, existing methods for monocular hand reconstruction typically rely on a weak perspective camera model, which simulates hand motion within a limited camera frustum. As a result, these approaches struggle to recover the full 3D global trajectory and often produce noisy or incorrect depth estimations, particularly when the video is captured by dynamic or moving cameras, which is common in egocentric scenarios. Our Dyn-HaMR consists of a multi-stage, multi-objective optimization pipeline, that factors in (i) simultaneous localization and mapping (SLAM) to robustly estimate relative camera motion, (ii) an interacting-hand prior for generative infilling and to refine the interaction dynamics, ensuring plausible recovery under (self-)occlusions, and (iii) hierarchical initialization through a combination of state-of-the-art hand tracking methods. Through extensive evaluations on both in-the-wild and indoor datasets, we show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery. This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras. Our project page is at https://dyn-hamr.github.io/.