Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration

TL;DR

This paper introduces a novel method for recovering camera-space 3D hand meshes from a single RGB image by dividing the task into root-relative and root recovery, leveraging semantic relations and silhouette projections for improved accuracy.

Contribution

It proposes a new pipeline that explicitly uses semantic relations among joints and silhouette projections to enhance camera-space 3D hand mesh recovery from monocular images.

Findings

Achieves state-of-the-art results on FreiHAND, RHD, and Human3.6M datasets.

Effectively recovers both root-relative and camera-space hand meshes.

Demonstrates robustness through extensive experiments.

Abstract

Recent years have witnessed significant progress in 3D hand mesh recovery. Nevertheless, because of the intrinsic 2D-to-3D ambiguity, recovering camera-space 3D information from a single RGB image remains challenging. To tackle this problem, we divide camera-space mesh recovery into two sub-tasks, i.e., root-relative mesh recovery and root recovery. First, joint landmarks and silhouette are extracted from a single input image to provide 2D cues for the 3D tasks. In the root-relative mesh recovery task, we exploit semantic relations among joints to generate a 3D mesh from the extracted 2D cues. Such generated 3D mesh coordinates are expressed relative to a root position, i.e., wrist of the hand. In the root recovery task, the root position is registered to the camera space by aligning the generated 3D mesh back to 2D cues, thereby completing cameraspace 3D mesh recovery. Our pipeline is novel in that (1) it explicitly makes use of known semantic relations among joints and (2) it exploits 1D projections of the silhouette and mesh to achieve robust registration. Extensive experiments on popular datasets such as FreiHAND, RHD, and Human3.6M demonstrate that our approach achieves stateof-the-art performance on both root-relative mesh recovery and root recovery. Our code is publicly available at https://github.com/SeanChenxy/HandMesh.