AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition

TL;DR

This paper proposes a novel AU-Expression Knowledge Constrained Representation Learning framework that leverages facial action unit correlations to improve facial expression recognition in uncontrolled environments without requiring AU annotations.

Contribution

It introduces a knowledge-guided attention mechanism and a correlation-based learning approach to enhance expression recognition without AU labels, outperforming current methods.

Findings

Outperforms state-of-the-art on uncontrolled datasets

Effectively learns AU representations without annotations

Enhances facial expression recognition accuracy

Abstract

Recognizing human emotion/expressions automatically is quite an expected ability for intelligent robotics, as it can promote better communication and cooperation with humans. Current deep-learning-based algorithms may achieve impressive performance in some lab-controlled environments, but they always fail to recognize the expressions accurately for the uncontrolled in-the-wild situation. Fortunately, facial action units (AU) describe subtle facial behaviors, and they can help distinguish uncertain and ambiguous expressions. In this work, we explore the correlations among the action units and facial expressions, and devise an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. Specifically, it leverages AU-expression correlations to guide the learning of the AU classifiers, and thus it can obtain AU representations without incurring any AU annotations. Then, it introduces a knowledge-guided attention mechanism that mines useful AU representations under the constraint of AU-expression correlations. In this way, the framework can capture local discriminative and complementary features to enhance facial representation for facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods. Codes and trained models are available at https://github.com/HCPLab-SYSU/AUE-CRL.