MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild

TL;DR

MaskHand introduces a generative masked modeling approach for 3D hand mesh reconstruction from a single RGB image, effectively handling ambiguities and occlusions to produce accurate and realistic results.

Contribution

The paper presents MaskHand, a novel probabilistic generative model that improves 3D hand mesh reconstruction by learning and sampling from the distribution of possible hand poses.

Findings

Achieves state-of-the-art accuracy on benchmark datasets

Demonstrates robustness to occlusions and ambiguities

Produces realistic and confident hand mesh reconstructions

Abstract

Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MaskHand, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MaskHand consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequence, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MaskHand achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: https://m-usamasaleem.github.io/publication/MaskHand/MaskHand.html.