Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

TL;DR

This paper introduces a two-stage gesture-aware pretraining framework that leverages gesture semantics to improve 3D hand pose estimation from monocular RGB images, demonstrating consistent accuracy gains.

Contribution

The work proposes a novel gesture-aware pretraining approach combined with token fusion for enhanced 3D hand pose estimation, transferable across architectures.

Findings

Gesture-aware pretraining improves accuracy over state-of-the-art baseline.

The method transfers benefits across different architectures without modification.

Experiments on InterHand2.6M validate the effectiveness of the approach.

Abstract

Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.