Multi-Task Learning Framework for Emotion Recognition in-the-wild

TL;DR

This paper introduces a multi-task learning system for emotion recognition in videos, combining unsupervised and supervised visual feature extraction with various temporal encoders to improve affective analysis.

Contribution

The paper proposes a novel multi-task learning framework integrating multiple feature extraction and temporal encoding methods for affective behavior analysis.

Findings

Achieved top performance in the MTL Challenge with a score of 1.7607 on validation.

Effectively modeled correlations between valence, arousal, expression, and AU tasks.

Demonstrated the effectiveness of combining unsupervised and supervised learning for visual features.

Abstract

This paper presents our system for the Multi-Task Learning (MTL) Challenge in the 4th Affective Behavior Analysis in-the-wild (ABAW) competition. We explore the research problems of this challenge from three aspects: 1) For obtaining efficient and robust visual feature representations, we propose MAE-based unsupervised representation learning and IResNet/DenseNet-based supervised representation learning methods; 2) Considering the importance of temporal information in videos, we explore three types of sequential encoders to capture the temporal information, including the encoder based on transformer, the encoder based on LSTM, and the encoder based on GRU; 3) For modeling the correlation between these different tasks (i.e., valence, arousal, expression, and AU) for multi-task affective analysis, we first explore the dependency between these different tasks and propose three multi-task learning frameworks to model the correlations effectively. Our system achieves the performance of $1.7607$ on the validation dataset and $1.4361$ on the test dataset, ranking first in the MTL Challenge. The code is available at https://github.com/AIM3-RUC/ABAW4.