Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction

TL;DR

This paper introduces a high-performance speech emotion recognition framework tailored for naturalistic conditions, utilizing multimodal and multi-task learning to address labeling disagreements and data imbalance, achieving top results in a challenge.

Contribution

The paper presents a reproducible framework that combines multimodal and multi-task learning techniques to improve SER performance in naturalistic settings, winning the IS25-SER Challenge.

Findings

Achieved top performance in the IS25-SER Challenge

Utilized multimodal and multi-task learning strategies

Ensemble of two systems yielded best results

Abstract

Speech emotion recognition (SER) in naturalistic conditions presents a significant challenge for the speech processing community. Challenges include disagreement in labeling among annotators and imbalanced data distributions. This paper presents a reproducible framework that achieves superior (top 1) performance in the Emotion Recognition in Naturalistic Conditions Challenge (IS25-SER Challenge) - Task 2, evaluated on the MSP-Podcast dataset. Our system is designed to tackle the aforementioned challenges through multimodal learning, multi-task learning, and imbalanced data handling. Specifically, our best system is trained by adding text embeddings, predicting gender, and including ``Other'' (O) and ``No Agreement'' (X) samples in the training set. Our system's results secured both first and second places in the IS25-SER Challenge, and the top performance was achieved by a simple two-system ensemble.