Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape

TL;DR

This paper introduces a novel method for high-fidelity 3D face shape reconstruction from a single image by leveraging multiview rendering, neural networks, and augmented ground-truth data to improve visual realism and detail.

Contribution

It proposes a comprehensive approach combining multiview normalization, a specialized neural network, and augmented ground-truth data to enhance 3D face shape reconstruction accuracy.

Findings

Achieves superior reconstruction accuracy on challenging datasets.

Generates detailed 3D shapes closely matching individual faces.

Outperforms existing 3DMM-based methods in visual verisimilitude.

Abstract

3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. However, previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry, which is attributed to insufficient ground-truth 3D shapes, unreliable training strategies and limited representation power of 3DMM. To alleviate this issue, this paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person. Specifically, given a 2D image as the input, we virtually render the image in several calibrated views to normalize pose variations while preserving the original image geometry. A many-to-one hourglass network serves as the encode-decoder to fuse multiview features and generate vertex displacements as the fine-grained geometry. Besides, the neural network is trained by directly optimizing the visual effect, where two 3D shapes are compared by measuring the similarity between the multiview images rendered from the shapes. Finally, we propose to generate the ground-truth 3D shapes by registering RGB-D images followed by pose and shape augmentation, providing sufficient data for network training. Experiments on several challenging protocols demonstrate the superior reconstruction accuracy of our proposal on the face shape.