UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment

TL;DR

UrgentMOS is a unified framework for speech quality assessment that leverages diverse metrics and preferences, improving robustness and performance across various speech datasets and evaluation scenarios.

Contribution

It introduces a novel multi-metric and preference learning approach that handles heterogeneous supervision and partial annotations for robust speech quality evaluation.

Findings

Achieves state-of-the-art results in absolute speech quality prediction.

Effectively models pairwise preferences for comparative evaluation.

Demonstrates robustness across diverse speech datasets and distortions.

Abstract

Automatic speech quality assessment has become increasingly important as modern speech generation systems continue to advance, while human listening tests remain costly, time-consuming, and difficult to scale. Most existing learning-based assessment models rely primarily on scarce human-annotated mean opinion score (MOS) data, which limits robustness and generalization, especially when training across heterogeneous datasets. In this work, we propose UrgentMOS, a unified speech quality assessment framework that jointly learns from diverse objective and perceptual quality metrics, while explicitly tolerating the absence of arbitrary subsets of metrics during training. By leveraging complementary quality facets under heterogeneous supervision, UrgentMOS enables effective utilization of partially annotated data and improves robustness when trained on large-scale, multi-source datasets. Beyond absolute score prediction, UrgentMOS explicitly models pairwise quality preferences by directly predicting comparative MOS (CMOS), making it well suited for preference-based evaluation scenarios commonly adopted in system benchmarking. Extensive experiments across a wide range of speech quality datasets, including simulated distortions, speech enhancement, and speech synthesis, demonstrate that UrgentMOS consistently achieves state-of-the-art performance in both absolute and comparative evaluation settings.