Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance

TL;DR

This paper presents a weakly-supervised 3D hand reconstruction method that leverages human hand knowledge and uncertainty modeling, enabling high performance without expensive 3D annotations.

Contribution

It introduces a novel approach that integrates biomechanical and physical insights into 3D hand reconstruction using differentiable losses and uncertainty modeling.

Findings

Achieves 21% performance improvement on FreiHAND dataset

Effectively incorporates hand knowledge into weakly-supervised learning

Outperforms existing state-of-the-art methods

Abstract

Fully-supervised monocular 3D hand reconstruction is often difficult because capturing the requisite 3D data entails deploying specialized equipment in a controlled environment. We introduce a weakly-supervised method that avoids such requirements by leveraging fundamental principles well-established in the understanding of the human hand's unique structure and functionality. Specifically, we systematically study hand knowledge from different sources, including biomechanics, functional anatomy, and physics. We effectively incorporate these valuable foundational insights into 3D hand reconstruction models through an appropriate set of differentiable training losses. This enables training solely with readily-obtainable 2D hand landmark annotations and eliminates the need for expensive 3D supervision. Moreover, we explicitly model the uncertainty that is inherent in image observations. We enhance the training process by exploiting a simple yet effective Negative Log Likelihood (NLL) loss that incorporates uncertainty into the loss function. Through extensive experiments, we demonstrate that our method significantly outperforms state-of-the-art weakly-supervised methods. For example, our method achieves nearly a 21\% performance improvement on the widely adopted FreiHAND dataset.