REALY: Rethinking the Evaluation of 3D Face Reconstruction

TL;DR

This paper introduces a new evaluation method and benchmark for 3D face reconstruction that uses region-wise shape alignment for more precise performance assessment, revealing detailed insights into state-of-the-art methods.

Contribution

It proposes a novel region-wise shape alignment evaluation approach and introduces the REALY benchmark with high-quality data for more accurate 3D face reconstruction assessment.

Findings

DECA excels on nose regions

GANFit performs better on cheek regions

The new evaluation reveals detailed performance differences

Abstract

The evaluation of 3D face reconstruction results typically relies on a rigid shape alignment between the estimated 3D model and the ground-truth scan. We observe that aligning two shapes with different reference points can largely affect the evaluation results. This poses difficulties for precisely diagnosing and improving a 3D face reconstruction method. In this paper, we propose a novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topology-consistent meshes. Our approach performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors. The fine-grained, region-wise evaluation results provide us detailed understandings about the performance of state-of-the-art 3D face reconstruction methods. For example, our experiments on single-image based reconstruction methods reveal that DECA performs the best on nose regions, while GANFit performs better on cheek regions. Besides, a new and high-quality 3DMM basis, HIFI3D++, is further derived using the same procedure as we construct REALY to align and retopologize several 3D face datasets. We will release REALY, HIFI3D++, and our new evaluation pipeline at https://realy3dface.com.