CLIP-Hand3D: Exploiting 3D Hand Pose Estimation via Context-Aware Prompting

TL;DR

This paper introduces CLIP-Hand3D, a novel 3D hand pose estimation method that leverages CLIP's contrastive learning to connect text prompts with detailed 3D hand joint features from monocular images, achieving state-of-the-art results.

Contribution

It proposes a new approach that bridges text prompts with 3D hand pose features using contrastive learning, enabling faster and more accurate hand pose estimation.

Findings

Achieves state-of-the-art performance on public benchmarks.

Provides significantly faster inference speed.

Effectively encodes pose-aware features from spatial joint distributions.

Abstract

Contrastive Language-Image Pre-training (CLIP) starts to emerge in many computer vision tasks and has achieved promising performance. However, it remains underexplored whether CLIP can be generalized to 3D hand pose estimation, as bridging text prompts with pose-aware features presents significant challenges due to the discrete nature of joint positions in 3D space. In this paper, we make one of the first attempts to propose a novel 3D hand pose estimator from monocular images, dubbed as CLIP-Hand3D, which successfully bridges the gap between text prompts and irregular detailed pose distribution. In particular, the distribution order of hand joints in various 3D space directions is derived from pose labels, forming corresponding text prompts that are subsequently encoded into text representations. Simultaneously, 21 hand joints in the 3D space are retrieved, and their spatial distribution (in x, y, and z axes) is encoded to form pose-aware features. Subsequently, we maximize semantic consistency for a pair of pose-text features following a CLIP-based contrastive learning paradigm. Furthermore, a coarse-to-fine mesh regressor is designed, which is capable of effectively querying joint-aware cues from the feature pyramid. Extensive experiments on several public hand benchmarks show that the proposed model attains a significantly faster inference speed while achieving state-of-the-art performance compared to methods utilizing the similar scale backbone.