3D Hand Pose Estimation in Everyday Egocentric Images

TL;DR

This paper investigates the challenges of 3D hand pose estimation in egocentric images, analyzing factors like perspective distortion and lack of annotations, and introduces WildHands, a system that outperforms existing methods across multiple datasets.

Contribution

The paper systematically studies the impact of various practices on 3D hand pose estimation in the wild and presents WildHands, a new system that achieves state-of-the-art results with less data and smaller size.

Findings

WildHands outperforms previous methods by 7.4% to 66% on diverse datasets.

Incorporating camera info and auxiliary supervision improves estimation accuracy.

WildHands is 10x smaller and trained on 5x less data than comparable systems.

Abstract

3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.