Image-free Domain Generalization via CLIP for 3D Hand Pose Estimation

TL;DR

This paper introduces an image-free domain generalization method for 3D hand pose estimation that leverages CLIP to incorporate text-based features, improving robustness without needing target domain images.

Contribution

The proposed approach uses CLIP to manipulate image features with text descriptions, enabling domain generalization without additional target domain data.

Findings

Outperforms state-of-the-art domain generalization methods on STB and RHD datasets.

Requires only source domain data, reducing data collection efforts.

Enhances robustness of hand pose estimation across diverse conditions.

Abstract

RGB-based 3D hand pose estimation has been successful for decades thanks to large-scale databases and deep learning. However, the hand pose estimation network does not operate well for hand pose images whose characteristics are far different from the training data. This is caused by various factors such as illuminations, camera angles, diverse backgrounds in the input images, etc. Many existing methods tried to solve it by supplying additional large-scale unconstrained/target domain images to augment data space; however collecting such large-scale images takes a lot of labors. In this paper, we present a simple image-free domain generalization approach for the hand pose estimation framework that uses only source domain data. We try to manipulate the image features of the hand pose estimation network by adding the features from text descriptions using the CLIP (Contrastive Language-Image Pre-training) model. The manipulated image features are then exploited to train the hand pose estimation network via the contrastive learning framework. In experiments with STB and RHD datasets, our algorithm shows improved performance over the state-of-the-art domain generalization approaches.