3D Dense Geometry-Guided Facial Expression Synthesis by Adversarial Learning

TL;DR

This paper introduces a novel GAN-based method for facial expression synthesis that leverages 3D dense geometry information, such as depth and surface normals, to improve manipulation accuracy without requiring paired training data.

Contribution

The authors propose a new GAN framework utilizing 3D dense geometry cues and a large-scale estimated RGB-Depth dataset, along with a confidence regulariser, to enhance facial expression synthesis.

Findings

Outperforms existing methods on AffectNet and RaFD benchmarks.

Effectively uses estimated depth information without ground truth depth maps.

Achieves significant improvements in expression manipulation quality.

Abstract

Manipulating facial expressions is a challenging task due to fine-grained shape changes produced by facial muscles and the lack of input-output pairs for supervised learning. Unlike previous methods using Generative Adversarial Networks (GAN), which rely on cycle-consistency loss or sparse geometry (landmarks) loss for expression synthesis, we propose a novel GAN framework to exploit 3D dense (depth and surface normals) information for expression manipulation. However, a large-scale dataset containing RGB images with expression annotations and their corresponding depth maps is not available. To this end, we propose to use an off-the-shelf state-of-the-art 3D reconstruction model to estimate the depth and create a large-scale RGB-Depth dataset after a manual data clean-up process. We utilise this dataset to minimise the novel depth consistency loss via adversarial learning (note we do not have ground truth depth maps for generated face images) and the depth categorical loss of synthetic data on the discriminator. In addition, to improve the generalisation and lower the bias of the depth parameters, we propose to use a novel confidence regulariser on the discriminator side of the framework. We extensively performed both quantitative and qualitative evaluations on two publicly available challenging facial expression benchmarks: AffectNet and RaFD. Our experiments demonstrate that the proposed method outperforms the competitive baseline and existing arts by a large margin.