The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition

TL;DR

MuSe 2024 is a multimodal challenge focusing on social perception and humor recognition, providing datasets and baselines to advance affective computing and sentiment analysis across diverse modalities and cultural contexts.

Contribution

This paper introduces two new sub-challenges and datasets for multimodal sentiment analysis, along with baseline models using Transformers and RNNs for social perception and humor detection.

Findings

Achieved a mean Pearson's ρ of 0.3573 on MuSe-Perception

Achieved an AUC of 0.8682 on MuSe-Humor

Established baseline systems for future research

Abstract

The Multimodal Sentiment Analysis Challenge (MuSe) 2024 addresses two contemporary multimodal affect and sentiment analysis problems: In the Social Perception Sub-Challenge (MuSe-Perception), participants will predict 16 different social attributes of individuals such as assertiveness, dominance, likability, and sincerity based on the provided audio-visual data. The Cross-Cultural Humor Detection Sub-Challenge (MuSe-Humor) dataset expands upon the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset, focusing on the detection of spontaneous humor in a cross-lingual and cross-cultural setting. The main objective of MuSe 2024 is to unite a broad audience from various research domains, including multimodal sentiment analysis, audio-visual affective computing, continuous signal processing, and natural language processing. By fostering collaboration and exchange among experts in these fields, the MuSe 2024 endeavors to advance the understanding and application of sentiment analysis and affective computing across multiple modalities. This baseline paper provides details on each sub-challenge and its corresponding dataset, extracted features from each data modality, and discusses challenge baselines. For our baseline system, we make use of a range of Transformers and expert-designed features and train Gated Recurrent Unit (GRU)-Recurrent Neural Network (RNN) models on them, resulting in a competitive baseline system. On the unseen test datasets of the respective sub-challenges, it achieves a mean Pearson's Correlation Coefficient ($\rho$) of 0.3573 for MuSe-Perception and an Area Under the Curve (AUC) value of 0.8682 for MuSe-Humor.