Bridging the Gaps: Utilizing Unlabeled Face Recognition Datasets to Boost Semi-Supervised Facial Expression Recognition

TL;DR

This paper introduces a semi-supervised facial expression recognition approach that leverages unlabeled face recognition datasets, face reconstruction pre-training, and a novel Mixup data augmentation to improve performance with limited labeled data.

Contribution

The authors propose a new semi-supervised FER method utilizing unlabeled face recognition data, face reconstruction pre-training, and a Mixup-based augmentation strategy, achieving state-of-the-art results.

Findings

Outperforms existing semi-supervised FER methods.

Achieves 64.02% accuracy on AffectNet with only 5% labeled data.

Achieves 88.23% accuracy on RAF-DB with 25% labeled data.

Abstract

In recent years, Facial Expression Recognition (FER) has gained increasing attention. Most current work focuses on supervised learning, which requires a large amount of labeled and diverse images, while FER suffers from the scarcity of large, diverse datasets and annotation difficulty. To address these problems, we focus on utilizing large unlabeled Face Recognition (FR) datasets to boost semi-supervised FER. Specifically, we first perform face reconstruction pre-training on large-scale facial images without annotations to learn features of facial geometry and expression regions, followed by two-stage fine-tuning on FER datasets with limited labels. In addition, to further alleviate the scarcity of labeled and diverse images, we propose a Mixup-based data augmentation strategy tailored for facial images, and the loss weights of real and virtual images are determined according to the intersection-over-union (IoU) of the faces in the two images. Experiments on RAF-DB, AffectNet, and FERPlus show that our method outperforms existing semi-supervised FER methods and achieves new state-of-the-art performance. Remarkably, with only 5%, 25% training sets,our method achieves 64.02% on AffectNet,and 88.23% on RAF-DB, which is comparable to fully supervised state-of-the-art methods. Codes will be made publicly available at https://github.com/zhelishisongjie/SSFER.