SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data

TL;DR

SynFER introduces a novel synthetic data generation framework for facial expression recognition, leveraging high-level textual descriptions and facial action units to improve model training without large real datasets.

Contribution

The paper presents SynFER, a new synthetic data generation method that uses semantic guidance and pseudo-label correction for high-quality facial expression images based on textual descriptions and facial action units.

Findings

Achieves 67.23% accuracy on AffectNet with synthetic data alone.

Scaling synthetic data to five times improves accuracy to 69.84%.

Demonstrates synthetic data's effectiveness for facial expression recognition.

Abstract

Facial expression datasets remain limited in scale due to the subjectivity of annotations and the labor-intensive nature of data collection. This limitation poses a significant challenge for developing modern deep learning-based facial expression analysis models, particularly foundation models, that rely on large-scale data for optimal performance. To tackle the overarching and complex challenge, instead of introducing a new large-scale dataset, we introduce SynFER (Synthesis of Facial Expressions with Refined Control), a novel synthetic framework for synthesizing facial expression image data based on high-level textual descriptions as well as more fine-grained and precise control through facial action units. To ensure the quality and reliability of the synthetic data, we propose a semantic guidance technique to steer the generation process and a pseudo-label generator to help rectify the facial expression labels for the synthetic images. To demonstrate the generation fidelity and the effectiveness of the synthetic data from SynFER, we conduct extensive experiments on representation learning using both synthetic data and real-world data. Results validate the efficacy of our approach and the synthetic data. Notably, our approach achieves a 67.23% classification accuracy on AffectNet when training solely with synthetic data equivalent to the AffectNet training set size, which increases to 69.84% when scaling up to five times the original size. Code is available here.