Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play

TL;DR

Speech-DRAME introduces a comprehensive framework for evaluating speech role-play models, combining new benchmarks, a fine-tuned evaluation model, and a role-play benchmark to improve assessment accuracy and reflect real-world nuances.

Contribution

The paper presents Speech-DRAME, a unified framework with new benchmarks and a specialized evaluation model that outperforms existing zero-shot methods in assessing speech role-play quality.

Findings

DRAME-Eval outperforms zero-shot ALLMs in correlation with human ratings.

Speech-DRAME provides the first comprehensive, reproducible foundation for speech role-play evaluation.

Benchmark resources and evaluation strategies improve assessment of nuanced speech qualities.

Abstract

Role-play has become a key testbed for generative models, expanding from text-only dialogue to multimodal interaction. Extending role-play to speech captures prosody, emotion, and delivery, but also poses new evaluation challenges. Current pipelines often use audio large language models (ALLMs) as zero-shot judges, which miss paralinguistic cues, collapse multiple aspects into coarse scores, and rely on synthetic speech references that fail to reflect real-world roles. We present Speech-DRAME, a unified framework that contributes at three levels: (i) Speech-DRAME-EvalBench, an evaluation benchmark with bilingual human-annotated data and protocols for training and testing speech evaluation models (SEMs), (ii) DRAME-Eval, a fine-tuned evaluation model, which substantially outperforms zero-shot and few-shot ALLMs, and (iii) Speech-DRAME-RoleBench, a speech role-play benchmark that leverages DRAME-Eval as an automatic judge to compare speech foundation models (SFMs). Speech-DRAME distinguishes between two complementary evaluation strategies: Archetype Evaluation, a top-down approach measuring adherence to broad role archetypes, and Realism Evaluation, a bottom-up approach grounded in real human speech that emphasizes nuanced role quality. Compared to zero-shot ALLM judges, DRAME-Eval achieves stronger agreement with human ratings (Pearson correlation from 0.480 to 0.629 in archetypes, and 0.390 to 0.625 in realism). By integrating transparent benchmark resources, modeling approaches, and system-level evaluation, Speech-DRAME provides the first comprehensive, reproducible foundation for assessing spoken role-play.