Mask2Hand: Learning to Predict the 3D Hand Pose and Shape from Shadow

TL;DR

Mask2Hand is a self-supervised approach that predicts 3D hand pose and shape from a single binary silhouette, eliminating the need for manual annotations and performing comparably to RGB/depth-based methods.

Contribution

It introduces a novel self-trainable framework using differentiable rendering to estimate 3D hand models solely from binary masks without extra labeled data.

Findings

Achieves comparable accuracy to RGB/depth methods using only binary masks.

Effectively constrains hand pose and shape through silhouette-based loss functions.

Operates in both aligned and unaligned settings with minimal input data.

Abstract

We present a self-trainable method, Mask2Hand, which learns to solve the challenging task of predicting 3D hand pose and shape from a 2D binary mask of hand silhouette/shadow without additional manually-annotated data. Given the intrinsic camera parameters and the parametric hand model in the camera space, we adopt the differentiable rendering technique to project 3D estimations onto the 2D binary silhouette space. By applying a tailored combination of losses between the rendered silhouette and the input binary mask, we are able to integrate the self-guidance mechanism into our end-to-end optimization process for constraining global mesh registration and hand pose estimation. The experiments show that our method, which takes a single binary mask as the input, can achieve comparable prediction accuracy on both unaligned and aligned settings as state-of-the-art methods that require RGB or depth inputs. Our code is available at https://github.com/lijenchang/Mask2Hand.