3D Hand Pose Tracking and Estimation Using Stereo Matching

TL;DR

This paper introduces a stereo matching-based approach for 3D hand pose tracking and estimation, providing a new benchmark dataset and a specialized hand segmentation algorithm that performs comparably to active depth sensors in various scenarios.

Contribution

It presents a novel stereo-based hand segmentation algorithm and a comprehensive benchmark dataset for 3D hand pose estimation using passive stereo matching.

Findings

Stereo matching algorithms maintain performance with correct hand segmentation.

The proposed segmentation algorithm is effective for hand tracking.

Tracking quality is comparable to active depth sensors in challenging scenarios.

Abstract

3D hand pose tracking/estimation will be very important in the next generation of human-computer interaction. Most of the currently available algorithms rely on low-cost active depth sensors. However, these sensors can be easily interfered by other active sources and require relatively high power consumption. As a result, they are currently not suitable for outdoor environments and mobile devices. This paper aims at tracking/estimating hand poses using passive stereo which avoids these limitations. A benchmark with 18,000 stereo image pairs and 18,000 depth images captured from different scenarios and the ground-truth 3D positions of palm and finger joints (obtained from the manual label) is thus proposed. This paper demonstrates that the performance of the state-of-the art tracking/estimation algorithms can be maintained with most stereo matching algorithms on the proposed benchmark, as long as the hand segmentation is correct. As a result, a novel stereo-based hand segmentation algorithm specially designed for hand tracking/estimation is proposed. The quantitative evaluation demonstrates that the proposed algorithm is suitable for the state-of-the-art hand pose tracking/estimation algorithms and the tracking quality is comparable to the use of active depth sensors under different challenging scenarios.