SUN Team's Contribution to ABAW 2024 Competition: Audio-visual Valence-Arousal Estimation and Expression Recognition

TL;DR

This paper presents audiovisual deep learning methods for in-the-wild emotion recognition, focusing on CNN-based architectures and fusion strategies, evaluated on the AffWild2 dataset for the ABAW 2024 challenge.

Contribution

It explores the effectiveness of fine-tuned CNN and PDEM architectures for audiovisual emotion recognition in real-world conditions, comparing various temporal and fusion strategies.

Findings

Demonstrated improved emotion recognition accuracy on AffWild2 dataset.

Compared different temporal modeling and fusion strategies for multimodal data.

Provided insights into the effectiveness of CNN and PDEM architectures in in-the-wild scenarios.

Abstract

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely 'in-the-wild' data. This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem. We particularly explore the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively. We compare alternative temporal modeling and fusion strategies using the embeddings from these multi-stage trained modality-specific Deep Neural Networks (DNN). We report results on the AffWild2 dataset under Affective Behavior Analysis in-the-Wild 2024 (ABAW'24) challenge protocol.