AmbER$^2$: Dual Ambiguity-Aware Emotion Recognition Applied to Speech and Text

TL;DR

This paper introduces AmbER$^2$, a novel dual ambiguity-aware framework for emotion recognition that explicitly models both rater and modality ambiguity, leading to improved performance on speech and text datasets.

Contribution

It proposes a dual ambiguity-aware approach with a teacher-student architecture that explicitly models rater and modality ambiguity in emotion recognition.

Findings

AmbER$^2$ outperforms baseline models in distributional fidelity.

Achieves state-of-the-art or superior results on IEMOCAP and MSP-Podcast datasets.

Modeling ambiguity benefits highly uncertain samples.

Abstract

Emotion recognition is inherently ambiguous, with uncertainty arising both from rater disagreement and from discrepancies across modalities such as speech and text. There is growing interest in modeling rater ambiguity using label distributions. However, modality ambiguity remains underexplored, and multimodal approaches often rely on simple feature fusion without explicitly addressing conflicts between modalities. In this work, we propose AmbER$^2$, a dual ambiguity-aware framework that simultaneously models rater-level and modality-level ambiguity through a teacher-student architecture with a distribution-wise training objective. Evaluations on IEMOCAP and MSP-Podcast show that AmbER$^2$ consistently improves distributional fidelity over conventional cross-entropy baselines and achieves performance competitive with, or superior to, recent state-of-the-art systems. For example, on IEMOCAP, AmbER$^2$ achieves relative improvements of 20.3% on Bhattacharyya coefficient (0.83 vs. 0.69), 13.6% on R$^2$ (0.67 vs. 0.59), 3.8% on accuracy (0.683 vs. 0.658), and 4.5% on F1 (0.675 vs. 0.646). Further analysis across ambiguity levels shows that explicitly modeling ambiguity is particularly beneficial for highly uncertain samples. These findings highlight the importance of jointly addressing rater and modality ambiguity when building robust emotion recognition systems.