RAFaRe: Learning Robust and Accurate Non-parametric 3D Face Reconstruction from Pseudo 2D&3D Pairs

TL;DR

This paper introduces a non-parametric approach for single-view 3D face reconstruction that leverages a large-scale pseudo 2D&3D dataset to improve accuracy and generalization over existing parametric methods.

Contribution

It presents a novel non-parametric method utilizing a large pseudo 2D&3D dataset and a hierarchical signed distance function for improved 3D face reconstruction accuracy.

Findings

Outperforms previous methods on FaceScape and MICC benchmarks.

Generalizes well to various appearances, poses, and environments.

Effectively predicts detailed 3D facial geometry.

Abstract

We propose a robust and accurate non-parametric method for single-view 3D face reconstruction (SVFR). While tremendous efforts have been devoted to parametric SVFR, a visible gap still lies between the result 3D shape and the ground truth. We believe there are two major obstacles: 1) the representation of the parametric model is limited to a certain face database; 2) 2D images and 3D shapes in the fitted datasets are distinctly misaligned. To resolve these issues, a large-scale pseudo 2D\&3D dataset is created by first rendering the detailed 3D faces, then swapping the face in the wild images with the rendered face. These pseudo 2D&3D pairs are created from publicly available datasets which eliminate the gaps between 2D and 3D data while covering diverse appearances, poses, scenes, and illumination. We further propose a non-parametric scheme to learn a well-generalized SVFR model from the created dataset, and the proposed hierarchical signed distance function turns out to be effective in predicting middle-scale and small-scale 3D facial geometry. Our model outperforms previous methods on FaceScape-wild/lab and MICC benchmarks and is well generalized to various appearances, poses, expressions, and in-the-wild environments. The code is released at http://github.com/zhuhao-nju/rafare .