Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction

TL;DR

This paper introduces a three-stage pipeline for real-time 3D hand-mesh reconstruction from RGB images, achieving high accuracy in hand pose, shape, and mesh-image alignment for augmented reality applications.

Contribution

A novel multi-stage approach decouples hand-mesh reconstruction into joint prediction, rough mesh estimation, and refinement, improving accuracy and real-time performance.

Findings

Outperforms state-of-the-art in hand-mesh and pose accuracy.

Achieves superior mesh-image alignment quality.

Demonstrates real-time AR application scenarios.

Abstract

3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR). However, this requires not only real-time speed and accurate hand pose and shape but also plausible mesh-image alignment. While existing works already achieve promising results, meeting all three requirements is very challenging. This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages: a joint stage to predict hand joints and segmentation; a mesh stage to predict a rough hand mesh; and a refine stage to fine-tune it with an offset mesh for mesh-image alignment. With careful design in the network structure and in the loss functions, we can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions. Extensive quantitative and qualitative results on benchmark datasets demonstrate that the quality of our results outperforms the state-of-the-art methods on hand-mesh/pose precision and hand-image alignment. In the end, we also showcase several real-time AR scenarios.