Towards Egocentric 3D Hand Pose Estimation in Unseen Domains

TL;DR

This paper introduces V-HPOT, a camera-agnostic, self-supervised approach for egocentric 3D hand pose estimation that significantly improves cross-domain performance without extensive training data.

Contribution

V-HPOT's key innovation is estimating normalized keypoint depth and applying self-supervised test-time optimization, enabling robust cross-domain hand pose estimation.

Findings

Achieves 71% reduction in mean pose error on H2O dataset.

Achieves 41% reduction in mean pose error on AssemblyHands dataset.

Outperforms all single-stage methods and rivals two-stage approaches with less data.

Abstract

We present V-HPOT, a novel approach for improving the cross-domain performance of 3D hand pose estimation from egocentric images across diverse, unseen domains. State-of-the-art methods demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception -- overfitting to specific camera intrinsics. Our method addresses this by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model's depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing approximately x3.5 to x14 less data.