EmoSURA: Towards Accurate Evaluation of Detailed and Long-Context Emotional Speech Captions

TL;DR

EmoSURA introduces an atomic verification framework for evaluating detailed emotional speech captions, improving correlation with human judgments over traditional metrics, especially for long-form descriptions.

Contribution

The paper presents EmoSURA, a novel evaluation method that decomposes captions into atomic units and verifies them against speech, along with SURABench, a new benchmark resource.

Findings

EmoSURA correlates positively with human judgments.

Traditional metrics negatively correlate with human assessments for long captions.

SURABench provides a standardized evaluation dataset.

Abstract

Recent advancements in speech captioning models have enabled the generation of rich, fine-grained captions for emotional speech. However, the evaluation of such captions remains a critical bottleneck: traditional N-gram metrics fail to capture semantic nuances, while LLM judges often suffer from reasoning inconsistency and context-collapse when processing long-form descriptions. In this work, we propose EmoSURA, a novel evaluation framework that shifts the paradigm from holistic scoring to atomic verification. EmoSURA decomposes complex captions into Atomic Perceptual Units, which are self-contained statements regarding vocal or emotional attributes, and employs an audio-grounded verification mechanism to validate each unit against the raw speech signal. Furthermore, we address the scarcity of standardized evaluation resources by introducing SURABench, a carefully balanced and stratified benchmark. Our experiments show that EmoSURA achieves a positive correlation with human judgments, offering a more reliable assessment for long-form captions compared to traditional metrics, which demonstrated negative correlations due to their sensitivity to caption length.