DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

TL;DR

This paper introduces DeltaDorsal, a novel method that improves egocentric hand pose estimation by leveraging dorsal hand features and a dual-stream delta encoder, significantly reducing errors in occluded scenarios.

Contribution

The paper presents a new dorsal feature-based approach with a dual-stream delta encoder that outperforms existing methods in occluded hand pose estimation.

Findings

Reduces MPJAE by 18% in occluded scenarios

Improves reliability of finger pinch and tap detection

Enables new interaction paradigms like isometric force detection

Abstract

The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers >= 50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface "click" without visible movement while minimizing model size.