DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Models

TL;DR

DiffHand introduces a diffusion model-based approach for monocular hand mesh reconstruction, effectively handling depth ambiguity and occlusion by modeling uncertainty and refining meshes through a denoising process.

Contribution

It is the first to apply diffusion models to hand mesh reconstruction, reformulating the task as a denoising process and introducing a cross-modality decoder for better vertex connectivity modeling.

Findings

Achieves 5.8mm PA-MPJPE on Freihand dataset

Achieves 4.98mm PA-MPJPE on DexYCB dataset

Outperforms state-of-the-art methods in accuracy

Abstract

Hand mesh reconstruction from the monocular image is a challenging task due to its depth ambiguity and severe occlusion, there remains a non-unique mapping between the monocular image and hand mesh. To address this, we develop DiffHand, the first diffusion-based framework that approaches hand mesh reconstruction as a denoising diffusion process. Our one-stage pipeline utilizes noise to model the uncertainty distribution of the intermediate hand mesh in a forward process. We reformulate the denoising diffusion process to gradually refine noisy hand mesh and then select mesh with the highest probability of being correct based on the image itself, rather than relying on 2D joints extracted beforehand. To better model the connectivity of hand vertices, we design a novel network module called the cross-modality decoder. Extensive experiments on the popular benchmarks demonstrate that our method outperforms the state-of-the-art hand mesh reconstruction approaches by achieving 5.8mm PA-MPJPE on the Freihand test set, 4.98mm PA-MPJPE on the DexYCB test set.