# 3D Hand Shape and Pose from Images in the Wild

## TL;DR

This paper introduces an end-to-end deep learning approach that accurately predicts 3D hand shape and pose from in-the-wild RGB images, leveraging a model-based decoder and weak supervision for improved generalization.

## Contribution

It presents the first deep learning method combining shape and pose prediction with a model-based decoder for in-the-wild images, achieving state-of-the-art results.

## Key findings

- State-of-the-art 3D hand pose prediction performance.
- Geometrically valid and plausible 3D reconstructions.
- Effective use of weak supervision with 2D and 3D annotations.

## Abstract

We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild.