Evaluating the Expressive Appropriateness of Speech in Rich Contexts

TL;DR

This paper introduces CEAEval, a novel framework and dataset for evaluating whether speech expressively matches its narrative context, addressing limitations of existing emotion-focused methods.

Contribution

It presents CEAEval, a comprehensive context-rich evaluation framework and the first Mandarin speech dataset with annotations for expressive appropriateness.

Findings

CEAEval-M outperforms existing speech evaluation systems.

CEAEval-D is the first dataset with narrative context and expressive annotations in Mandarin.

The framework effectively assesses speech appropriateness in narrative contexts.

Abstract

Evaluating expressive speech remains challenging, as existing methods mainly assess emotional intensity and overlook whether a speech sample is expressively appropriate for its contextual setting. This limitation hinders reliable evaluation of speech systems used in narrative-driven and interactive applications, such as audiobooks and conversational agents. We introduce CEAEval, a Context-rich framework for Evaluating Expressive Appropriateness in speech, which assesses whether a speech sample expressively aligns with the underlying communicative intent implied by its discourse-level narrative context. To support this task, we construct CEAEval-D, the first context-rich speech dataset with real human performances in Mandarin conversational speech, providing narrative descriptions together with fifteen dimensions of human annotations covering expressive attributes and expressive appropriateness. We further develop CEAEval-M, a model that integrates knowledge distillation, planner-based multi-model collaboration, adaptive audio attention bias, and reinforcement learning to perform context-rich expressive appropriateness evaluation. Experiments on a human-annotated test set demonstrate that CEAEval-M substantially outperforms existing speech evaluation and analysis systems.