HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioning

TL;DR

This paper introduces HandDGP, an end-to-end framework that predicts camera-space hand meshes from single RGB images by unifying previous separate stages and incorporating a differentiable global positioning module, improving accuracy and consistency.

Contribution

The main novelty is the end-to-end unification of hand mesh prediction and camera-space positioning using a differentiable module, along with an image rectification step to address scale-depth ambiguity.

Findings

Outperforms previous methods on three public benchmarks.

Effectively preserves contextual and scale information during prediction.

Reduces scale-depth ambiguity through image rectification.

Abstract

Predicting camera-space hand meshes from single RGB images is crucial for enabling realistic hand interactions in 3D virtual and augmented worlds. Previous work typically divided the task into two stages: given a cropped image of the hand, predict meshes in relative coordinates, followed by lifting these predictions into camera space in a separate and independent stage, often resulting in the loss of valuable contextual and scale information. To prevent the loss of these cues, we propose unifying these two stages into an end-to-end solution that addresses the 2D-3D correspondence problem. This solution enables back-propagation from camera space outputs to the rest of the network through a new differentiable global positioning module. We also introduce an image rectification step that harmonizes both the training dataset and the input image as if they were acquired with the same camera, helping to alleviate the inherent scale-depth ambiguity of the problem. We validate the effectiveness of our framework in evaluations against several baselines and state-of-the-art approaches across three public benchmarks.