Cross-modal Deep Variational Hand Pose Estimation

TL;DR

This paper introduces a cross-modal deep variational approach for 3D hand pose estimation that learns a unified latent space across multiple data modalities, improving accuracy and enabling generative synthesis.

Contribution

It proposes a novel variational framework that jointly optimizes across modalities, allowing semi-supervised learning and effective 3D hand pose estimation from RGB images.

Findings

Outperforms state-of-the-art in 3D hand pose estimation from RGB images.

Performs comparably to specialized methods on depth images.

The model can generate consistent hand configurations across modalities.

Abstract

The human hand moves in complex and high-dimensional ways, making estimation of 3D hand pose configurations from images alone a challenging task. In this work we propose a method to learn a statistical hand model represented by a cross-modal trained latent space via a generative deep neural network. We derive an objective function from the variational lower bound of the VAE framework and jointly optimize the resulting cross-modal KL-divergence and the posterior reconstruction objective, naturally admitting a training regime that leads to a coherent latent space across multiple modalities such as RGB images, 2D keypoint detections or 3D hand configurations. Additionally, it grants a straightforward way of using semi-supervision. This latent space can be directly used to estimate 3D hand poses from RGB images, outperforming the state-of-the art in different settings. Furthermore, we show that our proposed method can be used without changes on depth images and performs comparably to specialized methods. Finally, the model is fully generative and can synthesize consistent pairs of hand configurations across modalities. We evaluate our method on both RGB and depth datasets and analyze the latent space qualitatively.