Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models

TL;DR

This paper investigates using large audio-language models to generate synthetic annotations that improve emotion recognition in speech, addressing annotation scarcity and ambiguity issues.

Contribution

It introduces a framework for creating synthetic perceptual proxies with ALMs to augment human annotations in speech emotion recognition.

Findings

Synthetic annotations improve emotion distribution accuracy in low-ambiguity cases.

Augmentation benefits diminish for highly ambiguous emotions with high human disagreement.

Proposed DiME-Aug strategy addresses class imbalance and enables unbiased evaluation.

Abstract

Speech Emotion Recognition models typically use single categorical labels, overlooking the inherent ambiguity of human emotions. Ambiguous Emotion Recognition addresses this by representing emotions as probability distributions, but progress is limited by unreliable ground-truth distributions inferred from sparse human annotations. This paper explores whether Large Audio-Language Models (ALMs) can mitigate the annotation bottleneck by generating high-quality synthetic annotations. We introduce a framework leveraging ALMs to create Synthetic Perceptual Proxies, augmenting human annotations to improve ground-truth distribution reliability. We validate these proxies through statistical analysis of their alignment with human distributions and evaluate their impact by fine-tuning ALMs with the augmented emotion distributions. Furthermore, to address class imbalance and enable unbiased evaluation, we propose DiME-Aug, a Distribution-aware Multimodal Emotion Augmentation strategy. Experiments on IEMOCAP and MSP-Podcast show that synthetic annotations enhance emotion distribution, especially in low-ambiguity regions where annotation agreement is high. However, benefits diminish for highly ambiguous emotions with greater human disagreement. This work provides the first evidence that ALMs could address annotation scarcity in ambiguous emotion recognition, but highlights the need for more advanced prompting or generation strategies to handle highly ambiguous cases.