Perspective Reconstruction of Human Faces by Joint Mesh and Landmark Regression

TL;DR

This paper introduces a method for 3D human face reconstruction that jointly predicts 3D mesh and 2D landmarks to improve accuracy under perspective projection, especially when faces are close to the camera.

Contribution

It proposes a novel joint regression approach for 3D mesh and landmarks, enabling accurate perspective reconstruction and pose estimation.

Findings

Achieved 1st place in ECCV 2022 WCPA challenge

Model is robust across different identities, expressions, and poses

Facilitates future research with released code and models

Abstract

Even though 3D face reconstruction has achieved impressive progress, most orthogonal projection-based face reconstruction methods can not achieve accurate and consistent reconstruction results when the face is very close to the camera due to the distortion under the perspective projection. In this paper, we propose to simultaneously reconstruct 3D face mesh in the world space and predict 2D face landmarks on the image plane to address the problem of perspective 3D face reconstruction. Based on the predicted 3D vertices and 2D landmarks, the 6DoF (6 Degrees of Freedom) face pose can be easily estimated by the PnP solver to represent perspective projection. Our approach achieves 1st place on the leader-board of the ECCV 2022 WCPA challenge and our model is visually robust under different identities, expressions and poses. The training code and models are released to facilitate future research.