Pixel-Aligned Non-parametric Hand Mesh Reconstruction

TL;DR

This paper introduces a simple, end-to-end neural network architecture for 3D hand mesh reconstruction from images, leveraging pixel-aligned features and hierarchical decoding to improve accuracy.

Contribution

The proposed method combines local 2D image features with 3D geometric features using a novel mesh decoder, achieving state-of-the-art results in hand mesh reconstruction.

Findings

Achieves state-of-the-art accuracy on FreiHAND dataset.

Effectively combines multi-scale features for detailed mesh recovery.

Utilizes a coarse-to-fine decoding strategy for improved mesh detail.

Abstract

Non-parametric mesh reconstruction has recently shown significant progress in 3D hand and body applications. In these methods, mesh vertices and edges are visible to neural networks, enabling the possibility to establish a direct mapping between 2D image pixels and 3D mesh vertices. In this paper, we seek to establish and exploit this mapping with a simple and compact architecture. The network is designed with these considerations: 1) aggregating both local 2D image features from the encoder and 3D geometric features captured in the mesh decoder; 2) decoding coarse-to-fine meshes along the decoding layers to make the best use of the hierarchical multi-scale information. Specifically, we propose an end-to-end pipeline for hand mesh recovery tasks which consists of three phases: a 2D feature extractor constructing multi-scale feature maps, a feature mapping module transforming local 2D image features to 3D vertex features via 3D-to-2D projection, and a mesh decoder combining the graph convolution and self-attention to reconstruct mesh. The decoder aggregate both local image features in pixels and geometric features in vertices. It also regresses the mesh vertices in a coarse-to-fine manner, which can leverage multi-scale information. By exploiting the local connection and designing the mesh decoder, Our approach achieves state-of-the-art for hand mesh reconstruction on the public FreiHAND dataset.