BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks

TL;DR

BiHand is an end-to-end model that recovers detailed 3D hand meshes from a single RGB image using a novel multi-stage bisected hourglass network architecture, improving robustness and accuracy.

Contribution

Introduces a multi-stage bisecting design in a cascaded network for joint 2D, 3D, and mesh hand estimation from RGB images, enhancing performance and robustness.

Findings

Achieves superior accuracy on RHD and STB benchmarks.

Produces high-quality 3D hand meshes in challenging conditions.

Outperforms state-of-the-art methods in hand mesh recovery.

Abstract

3D hand estimation has been a long-standing research topic in computer vision. A recent trend aims not only to estimate the 3D hand joint locations but also to recover the mesh model. However, achieving those goals from a single RGB image remains challenging. In this paper, we introduce an end-to-end learnable model, BiHand, which consists of three cascaded stages, namely 2D seeding stage, 3D lifting stage, and mesh generation stage. At the output of BiHand, the full hand mesh will be recovered using the joint rotations and shape parameters predicted from the network. Inside each stage, BiHand adopts a novel bisecting design which allows the networks to encapsulate two closely related information (e.g. 2D keypoints and silhouette in 2D seeding stage, 3D joints, and depth map in 3D lifting stage, joint rotations and shape parameters in the mesh generation stage) in a single forward pass. As the information represents different geometry or structure details, bisecting the data flow can facilitate optimization and increase robustness. For quantitative evaluation, we conduct experiments on two public benchmarks, namely the Rendered Hand Dataset (RHD) and the Stereo Hand Pose Tracking Benchmark (STB). Extensive experiments show that our model can achieve superior accuracy in comparison with state-of-the-art methods, and can produce appealing 3D hand meshes in several severe conditions.