Facial Action Units Detection Aided by Global-Local Expression Embedding

TL;DR

This paper introduces GLEE-Net, a novel framework that leverages expression datasets without AU labels to improve facial Action Unit detection by extracting identity-independent features through global and local embeddings, and consolidating them with 3D reconstruction and Transformer-based classification.

Contribution

The paper proposes a new AU detection method utilizing expression datasets without AU labels, combining global-local embeddings, 3D face reconstruction, and Transformer-based classification to address identity overfitting.

Findings

Outperforms state-of-the-art on DISFA, BP4D, and BP4D+ datasets.

Effectively alleviates identity overfitting in AU detection.

Utilizes expression datasets without AU labels for improved detection.

Abstract

Since Facial Action Unit (AU) annotations require domain expertise, common AU datasets only contain a limited number of subjects. As a result, a crucial challenge for AU detection is addressing identity overfitting. We find that AUs and facial expressions are highly associated, and existing facial expression datasets often contain a large number of identities. In this paper, we aim to utilize the expression datasets without AU labels to facilitate AU detection. Specifically, we develop a novel AU detection framework aided by the Global-Local facial Expressions Embedding, dubbed GLEE-Net. Our GLEE-Net consists of three branches to extract identity-independent expression features for AU detection. We introduce a global branch for modeling the overall facial expression while eliminating the impacts of identities. We also design a local branch focusing on specific local face regions. The combined output of global and local branches is firstly pre-trained on an expression dataset as an identity-independent expression embedding, and then finetuned on AU datasets. Therefore, we significantly alleviate the issue of limited identities. Furthermore, we introduce a 3D global branch that extracts expression coefficients through 3D face reconstruction to consolidate 2D AU descriptions. Finally, a Transformer-based multi-label classifier is employed to fuse all the representations for AU detection. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art on the widely-used DISFA, BP4D and BP4D+ datasets.