Semantic-aware One-shot Face Re-enactment with Dense Correspondence Estimation

TL;DR

This paper introduces a semantic-aware one-shot face re-enactment method utilizing 3D Morphable Models for explicit facial semantic decomposition and dense correspondence estimation, improving identity preservation and visual quality.

Contribution

It proposes a novel approach combining 3DMM-based semantic decomposition with dense correspondence estimation for more accurate and identity-preserving face re-enactment.

Findings

Outperforms existing methods in identity preservation.

Achieves higher visual quality in re-enactment results.

Demonstrates robustness across various datasets.

Abstract

One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces. Specifically, the suboptimally disentangled identity information of driving subjects would inevitably interfere with the re-enactment results and lead to face shape distortion. To solve this problem, this paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement. Instead of using 3D coefficients alone for re-enactment control, we take the advantage of the generative ability of 3DMM to render textured face proxies. These proxies contain abundant yet compact geometric and semantic information of human faces, which enable us to compute the face motion field between source and driving images by estimating the dense correspondence. In this way, we could approximate re-enactment results by warping source images according to the motion field, and a Generative Adversarial Network (GAN) is adopted to further improve the visual quality of warping results. Extensive experiments on various datasets demonstrate the advantages of the proposed method over existing start-of-the-art benchmarks in both identity preservation and re-enactment fulfillment.