MEDUSA: A Multimodal Deep Fusion Multi-Stage Training Framework for Speech Emotion Recognition in Naturalistic Conditions

TL;DR

MEDUSA is a comprehensive multimodal framework for speech emotion recognition that effectively manages class imbalance and emotion ambiguity through a four-stage training process involving ensemble classifiers, a novel fusion mechanism, and meta-classification.

Contribution

It introduces MEDUSA, a novel multi-stage training framework with a deep cross-modal transformer and soft target learning for naturalistic speech emotion recognition.

Findings

Ranked 1st in Interspeech 2025 SER Challenge.

Effective handling of class imbalance and emotion ambiguity.

Utilizes a novel deep fusion mechanism and multi-stage training.

Abstract

SER is a challenging task due to the subjective nature of human emotions and their uneven representation under naturalistic conditions. We propose MEDUSA, a multimodal framework with a four-stage training pipeline, which effectively handles class imbalance and emotion ambiguity. The first two stages train an ensemble of classifiers that utilize DeepSER, a novel extension of a deep cross-modal transformer fusion mechanism from pretrained self-supervised acoustic and linguistic representations. Manifold MixUp is employed for further regularization. The last two stages optimize a trainable meta-classifier that combines the ensemble predictions. Our training approach incorporates human annotation scores as soft targets, coupled with balanced data sampling and multitask learning. MEDUSA ranked 1st in Task 1: Categorical Emotion Recognition in the Interspeech 2025: Speech Emotion Recognition in Naturalistic Conditions Challenge.