Weakly Supervised Regional and Temporal Learning for Facial Action Unit Recognition

TL;DR

This paper introduces a weakly supervised framework for facial action unit recognition that leverages regional and temporal properties through self-supervised auxiliary tasks, significantly improving performance on benchmark datasets.

Contribution

It proposes a novel end-to-end weakly supervised learning framework that incorporates regional and temporal auxiliary tasks for better AU recognition.

Findings

Achieves state-of-the-art results on BP4D and DISFA datasets.

Effectively captures regional features, relations, and motion cues of AUs.

Demonstrates the effectiveness of self-supervised auxiliary tasks in AU recognition.

Abstract

Automatic facial action unit (AU) recognition is a challenging task due to the scarcity of manual annotations. To alleviate this problem, a large amount of efforts has been dedicated to exploiting various weakly supervised methods which leverage numerous unlabeled data. However, many aspects with regard to some unique properties of AUs, such as the regional and relational characteristics, are not sufficiently explored in previous works. Motivated by this, we take the AU properties into consideration and propose two auxiliary AU related tasks to bridge the gap between limited annotations and the model performance in a self-supervised manner via the unlabeled data. Specifically, to enhance the discrimination of regional features with AU relation embedding, we design a task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a single image based optical flow estimation task is proposed to leverage the dynamic change of facial muscles and encode the motion information into the global feature representation. Based on these two self-supervised auxiliary tasks, local features, mutual relation and motion cues of AUs are better captured in the backbone network. Furthermore, by incorporating semi-supervised learning, we propose an end-to-end trainable framework named weakly supervised regional and temporal learning (WSRTL) for AU recognition. Extensive experiments on BP4D and DISFA demonstrate the superiority of our method and new state-of-the-art performances are achieved.