Skeleton-aware multi-scale heatmap regression for 2D hand pose estimation

TL;DR

This paper introduces a skeleton-aware multi-scale heatmap regression framework for 2D hand pose estimation from RGB images, improving accuracy across different hand sizes and complex scenarios.

Contribution

It proposes a novel deep learning framework combining skeleton detection and multi-scale heatmap regression, with a new dataset for hand detection and pose estimation.

Findings

Outperforms state-of-the-art methods in accuracy

Effective in cluttered images and complex poses

Validated on two datasets with qualitative and quantitative results

Abstract

Existing RGB-based 2D hand pose estimation methods learn the joint locations from a single resolution, which is not suitable for different hand sizes. To tackle this problem, we propose a new deep learning-based framework that consists of two main modules. The former presents a segmentation-based approach to detect the hand skeleton and localize the hand bounding box. The second module regresses the 2D joint locations through a multi-scale heatmap regression approach that exploits the predicted hand skeleton as a constraint to guide the model. Furthermore, we construct a new dataset that is suitable for both hand detection and pose estimation. We qualitatively and quantitatively validate our method on two datasets. Results demonstrate that the proposed method outperforms state-of-the-art and can recover the pose even in cluttered images and complex poses.