P.808 Multilingual Speech Enhancement Testing: Approach and Results of URGENT 2025 Challenge

TL;DR

This paper reviews the ITU-T P.808 subjective testing method, proposes localization improvements for multilingual speech enhancement evaluation, analyzes URGENT 2025 Challenge results, and discusses the reliability of subjective metrics in the era of generative AI.

Contribution

It introduces a novel process for localizing text and audio in crowdsourced subjective tests and provides insights into the reliability of subjective metrics for generative speech enhancement methods.

Findings

Subjective (ACR MOS) and objective metrics should be complemented with phone fidelity metrics for generative SE.

Analysis of URGENT Challenge results reveals issues with current subjective testing reliability.

Localization scripts and methods will be released for multilingual speech enhancement evaluation.

Abstract

In speech quality estimation for speech enhancement (SE) systems, subjective listening tests so far are considered as the gold standard. This should be even more true considering the large influx of new generative or hybrid methods into the field, revealing issues of some objective metrics. Efforts such as the Interspeech 2025 URGENT Speech Enhancement Challenge also involving non-English datasets add the aspect of multilinguality to the testing procedure. In this paper, we provide a brief recap of the ITU-T P.808 crowdsourced subjective listening test method. A first novel contribution is our proposed process of localizing both text and audio components of Naderi and Cutler's implementation of crowdsourced subjective absolute category rating (ACR) listening tests involving text-to-speech (TTS). Further, we provide surprising analyses of and insights into URGENT Challenge results, tackling the reliability of (P.808) ACR subjective testing as gold standard in the age of generative AI. Particularly, it seems that for generative SE methods, subjective (ACR MOS) and objective (DNSMOS, NISQA) reference-free metrics should be accompanied by objective phone fidelity metrics to reliably detect hallucinations. Finally, we will soon release our localization scripts and methods for easy deployment for new multilingual speech enhancement subjective evaluations according to ITU-T P.808.