Recognizing Facial Expressions of Occluded Faces using Convolutional Neural Networks

TL;DR

This paper introduces a CNN-based method for recognizing facial expressions in occluded faces, specifically VR headset scenarios, by training on artificially occluded faces, achieving significant accuracy improvements over baseline models.

Contribution

The study proposes a novel training approach that enhances CNN performance on occluded faces by focusing on lower-face features, validated on FER+ and AffectNet datasets.

Findings

CNN models trained on occluded faces outperform baseline models by up to 13%

Lower face features contain sufficient information for accurate expression recognition

Models trained on occluded faces maintain high accuracy even with severe occlusion

Abstract

In this paper, we present an approach based on convolutional neural networks (CNNs) for facial expression recognition in a difficult setting with severe occlusions. More specifically, our task is to recognize the facial expression of a person wearing a virtual reality (VR) headset which essentially occludes the upper part of the face. In order to accurately train neural networks for this setting, in which faces are severely occluded, we modify the training examples by intentionally occluding the upper half of the face. This forces the neural networks to focus on the lower part of the face and to obtain better accuracy rates than models trained on the entire faces. Our empirical results on two benchmark data sets, FER+ and AffectNet, show that our CNN models' predictions on lower-half faces are up to 13% higher than the baseline CNN models trained on entire faces, proving their suitability for the VR setting. Furthermore, our models' predictions on lower-half faces are no more than 10% under the baseline models' predictions on full faces, proving that there are enough clues in the lower part of the face to accurately predict facial expressions.