Coherence Constraints in Facial Expression Recognition

TL;DR

This paper explores the use of coherence constraints in training CNN-based facial expression recognizers, leveraging semi-supervised learning with video data to improve accuracy and robustness, especially under occlusions.

Contribution

It introduces coherence constraints in semi-supervised CNN training for facial expression recognition, enhancing performance by exploiting temporal and part-based prediction consistency.

Findings

Coherence constraints improve recognition accuracy.

Shape-based predictors outperform appearance-based ones with occlusions.

Semi-supervised training effectively utilizes unlabeled video data.

Abstract

Recognizing facial expressions from static images or video sequences is a widely studied but still challenging problem. The recent progresses obtained by deep neural architectures, or by ensembles of heterogeneous models, have shown that integrating multiple input representations leads to state-of-the-art results. In particular, the appearance and the shape of the input face, or the representations of some face parts, are commonly used to boost the quality of the recognizer. This paper investigates the application of Convolutional Neural Networks (CNNs) with the aim of building a versatile recognizer of expressions in static images that can be further applied to video sequences. We first study the importance of different face parts in the recognition task, focussing on appearance and shape-related features. Then we cast the learning problem in the Semi-Supervised setting, exploiting video data, where only a few frames are supervised. The unsupervised portion of the training data is used to enforce three types of coherence, namely temporal coherence, coherence among the predictions on the face parts and coherence between appearance and shape-based representation. Our experimental analysis shows that coherence constraints can improve the quality of the expression recognizer, thus offering a suitable basis to profitably exploit unsupervised video sequences. Finally we present some examples with occlusions where the shape-based predictor performs better than the appearance one.