Self-Supervised Regional and Temporal Auxiliary Tasks for Facial Action Unit Recognition

TL;DR

This paper introduces a self-supervised learning framework for facial action unit recognition that leverages regional and temporal auxiliary tasks to improve performance using unlabeled data.

Contribution

It proposes two novel self-supervised auxiliary tasks, RoI inpainting and optical flow estimation, to better capture regional, relational, and motion features of AUs.

Findings

Achieves state-of-the-art results on BP4D and DISFA datasets.

Effectively captures regional features and motion cues of facial AUs.

Enhances model performance with limited labeled data.

Abstract

Automatic facial action unit (AU) recognition is a challenging task due to the scarcity of manual annotations. To alleviate this problem, a large amount of efforts has been dedicated to exploiting various methods which leverage numerous unlabeled data. However, many aspects with regard to some unique properties of AUs, such as the regional and relational characteristics, are not sufficiently explored in previous works. Motivated by this, we take the AU properties into consideration and propose two auxiliary AU related tasks to bridge the gap between limited annotations and the model performance in a self-supervised manner via the unlabeled data. Specifically, to enhance the discrimination of regional features with AU relation embedding, we design a task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a single image based optical flow estimation task is proposed to leverage the dynamic change of facial muscles and encode the motion information into the global feature representation. Based on these two self-supervised auxiliary tasks, local features, mutual relation and motion cues of AUs are better captured in the backbone network with the proposed regional and temporal based auxiliary task learning (RTATL) framework. Extensive experiments on BP4D and DISFA demonstrate the superiority of our method and new state-of-the-art performances are achieved.