Im2Mesh GAN: Accurate 3D Hand Mesh Recovery from a Single RGB Image

TL;DR

This paper introduces Im2Mesh GAN, a novel approach for recovering accurate 3D hand meshes directly from single RGB images using end-to-end adversarial training and graph-based mesh modeling.

Contribution

It presents a new GAN architecture that learns 3D hand meshes directly from images without relying on parametric models, capturing topological and surface features.

Findings

Outperforms state-of-the-art methods in 3D hand mesh recovery

Effective in both supervised and unsupervised settings

Captures detailed 3D surface features

Abstract

This work addresses hand mesh recovery from a single RGB image. In contrast to most of the existing approaches where the parametric hand models are employed as the prior, we show that the hand mesh can be learned directly from the input image. We propose a new type of GAN called Im2Mesh GAN to learn the mesh through end-to-end adversarial training. By interpreting the mesh as a graph, our model is able to capture the topological relationship among the mesh vertices. We also introduce a 3D surface descriptor into the GAN architecture to further capture the 3D features associated. We experiment two approaches where one can reap the benefits of coupled groundtruth data availability of images and the corresponding meshes, while the other combats the more challenging problem of mesh estimations without the corresponding groundtruth. Through extensive evaluations we demonstrate that the proposed method outperforms the state-of-the-art.