Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

TL;DR

This paper introduces a synergy-based method combining 3D Morphable Models and 3D facial landmarks to improve the accuracy and robustness of 3D facial geometry prediction, using a bidirectional representation cycle.

Contribution

It proposes a novel synergy process that leverages the relationship between 3DMM parameters and landmarks, enhancing 3D facial geometry estimation.

Findings

Achieves superior accuracy in facial geometry prediction

Demonstrates robustness across various scenarios

Utilizes simple network operations for fast inference

Abstract

This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction. Codes and data: https://choyingw.github.io/works/SynergyNet/