Which Evaluation for Which Model? A Taxonomy for Speech Model Assessment

TL;DR

This paper introduces a comprehensive taxonomy for evaluating speech models, aligning evaluation methods with model capabilities and task requirements to improve assessment consistency and identify gaps.

Contribution

It presents a unified, axes-based taxonomy for classifying speech model evaluations, facilitating better alignment between models and appropriate assessment protocols.

Findings

Classifies existing evaluations along three orthogonal axes.

Maps evaluations to model capabilities and methodological demands.

Identifies gaps in current evaluation coverage, such as prosody and reasoning.

Abstract

Speech foundation models have recently achieved remarkable capabilities across a wide range of tasks. However, their evaluation remains disjointed across tasks and model types. Different models excel at distinct aspects of speech processing and thus require different evaluation protocols. This paper proposes a unified taxonomy that addresses the question: Which evaluation is appropriate for which model? The taxonomy defines three orthogonal axes: the evaluation aspect being measured, the model capabilities required to attempt the task, and the task or protocol requirements needed to perform it. We classify a broad set of existing evaluations and benchmarks along these axes, spanning areas such as representation learning, speech generation, and interactive dialogue. By mapping each evaluation to the capabilities a model exposes (e.g., speech generation, real-time processing) and to its methodological demands (e.g., fine-tuning data, human judgment), the taxonomy provides a principled framework for aligning models with suitable evaluation methods. It also reveals systematic gaps, such as limited coverage of prosody, interaction, or reasoning, that highlight priorities for future benchmark design. Overall, this work offers a conceptual foundation and practical guide for selecting, interpreting, and extending evaluations of speech models.