Affective Behaviour Analysis Using Pretrained Model with Facial Priori

TL;DR

This paper introduces a novel affective behaviour analysis method leveraging pretrained facial models, combining Masked Auto-Encoder and Vision Transformer with AffectNet CNN, to improve emotion recognition accuracy and reduce annotation effort.

Contribution

It proposes a multi-task framework using pretrained models and a co-training strategy with independent views for more effective emotion analysis.

Findings

Effective on ABAW4 dataset

Improved valence-arousal regression accuracy

Demonstrated robustness of the co-training approach

Abstract

Affective behaviour analysis has aroused researchers' attention due to its broad applications. However, it is labor exhaustive to obtain accurate annotations for massive face images. Thus, we propose to utilize the prior facial information via Masked Auto-Encoder (MAE) pretrained on unlabeled face images. Furthermore, we combine MAE pretrained Vision Transformer (ViT) and AffectNet pretrained CNN to perform multi-task emotion recognition. We notice that expression and action unit (AU) scores are pure and intact features for valence-arousal (VA) regression. As a result, we utilize AffectNet pretrained CNN to extract expression scores concatenating with expression and AU scores from ViT to obtain the final VA features. Moreover, we also propose a co-training framework with two parallel MAE pretrained ViT for expression recognition tasks. In order to make the two views independent, we random mask most patches during the training process. Then, JS divergence is performed to make the predictions of the two views as consistent as possible. The results on ABAW4 show that our methods are effective.