Efficiently Creating 3D Training Data for Fine Hand Pose Estimation

TL;DR

This paper introduces a semi-automated approach to efficiently generate large, accurate 3D hand pose datasets from depth videos, significantly improving training data quality for hand pose estimation.

Contribution

A novel semi-automated labeling method that reduces manual effort and enhances accuracy in creating 3D hand pose datasets from depth videos.

Findings

Enhanced dataset quality improves hand pose estimation accuracy.

Method reduces manual labeling effort significantly.

Training with new data outperforms previous models.

Abstract

While many recent hand pose estimation methods critically rely on a training set of labelled frames, the creation of such a dataset is a challenging task that has been overlooked so far. As a result, existing datasets are limited to a few sequences and individuals, with limited accuracy, and this prevents these methods from delivering their full potential. We propose a semi-automated method for efficiently and accurately labeling each frame of a hand depth video with the corresponding 3D locations of the joints: The user is asked to provide only an estimate of the 2D reprojections of the visible joints in some reference frames, which are automatically selected to minimize the labeling work by efficiently optimizing a sub-modular loss function. We then exploit spatial, temporal, and appearance constraints to retrieve the full 3D poses of the hand over the complete sequence. We show that this data can be used to train a recent state-of-the-art hand pose estimation method, leading to increased accuracy. The code and dataset can be found on our website https://cvarlab.icg.tugraz.at/projects/hand_detection/