Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions

TL;DR

This paper introduces a GAN-based method for synthesizing facial expressions from 3D face reconstructions to augment training data, significantly improving facial action detection accuracy over traditional methods.

Contribution

The study presents a novel approach using 3D face reconstruction and GANs to generate synthetic facial expressions for enhanced classifier training.

Findings

Synthetic data improved detection accuracy.

Outperformed existing state-of-the-art methods.

Training on synthetic expressions outperformed training on real data.

Abstract

Critical obstacles in training classifiers to detect facial actions are the limited sizes of annotated video databases and the relatively low frequencies of occurrence of many actions. To address these problems, we propose an approach that makes use of facial expression generation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. To evaluate this approach, a deep neural network was trained on two separate datasets: One network was trained on video of synthesized facial expressions generated from FERA17; the other network was trained on unaltered video from the same database. Both networks used the same train and validation partitions and were tested on the test partition of actual video from FERA17. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.