Confidence-Weighted Local Expression Predictions for Occlusion Handling in Expression Recognition and Action Unit detection

TL;DR

This paper introduces a confidence-weighted local expression prediction method using Random Forests and autoencoders to improve facial expression recognition and action unit detection, especially under occlusions.

Contribution

It proposes a novel LEP representation combined with confidence weighting via autoencoders, enhancing occlusion robustness and prediction accuracy in FER systems.

Findings

LEP representation effectively describes facial expressions and AUs.

Confidence weighting improves robustness to occlusions.

Method outperforms existing approaches in occlusion scenarios.

Abstract

Fully-Automatic Facial Expression Recognition (FER) from still images is a challenging task as it involves handling large interpersonal morphological differences, and as partial occlusions can occasionally happen. Furthermore, labelling expressions is a time-consuming process that is prone to subjectivity, thus the variability may not be fully covered by the training data. In this work, we propose to train Random Forests upon spatially defined local subspaces of the face. The output local predictions form a categorical expression-driven high-level representation that we call Local Expression Predictions (LEPs). LEPs can be combined to describe categorical facial expressions as well as Action Units (AUs). Furthermore, LEPs can be weighted by confidence scores provided by an autoencoder network. Such network is trained to locally capture the manifold of the non-occluded training data in a hierarchical way. Extensive experiments show that the proposed LEP representation yields high descriptive power for categorical expressions and AU occurrence prediction, and leads to interesting perspectives towards the design of occlusion-robust and confidence-aware FER systems.