Multi-dimensional Speech Quality Assessment in Crowdsourcing

TL;DR

This paper develops and validates a multi-dimensional speech quality assessment tool for crowdsourcing, extending existing standards to include reverberation and overall quality, demonstrating its accuracy and utility in speech enhancement evaluation.

Contribution

It introduces a crowdsourcing implementation of a multi-dimensional speech quality assessment tool based on ITU standards, extending it to new dimensions and validating its effectiveness.

Findings

The tool is accurate and reproducible.

It effectively measures multiple speech quality dimensions.

Demonstrated utility in the ICASSP 2023 challenge.

Abstract

Subjective speech quality assessment is the gold standard for evaluating speech enhancement processing and telecommunication systems. The commonly used standard ITU-T Rec. P.800 defines how to measure speech quality in lab environments, and ITU-T Rec.~P.808 extended it for crowdsourcing. ITU-T Rec. P.835 extends P.800 to measure the quality of speech in the presence of noise. ITU-T Rec. P.804 targets the conversation test and introduces perceptual speech quality dimensions which are measured during the listening phase of the conversation. The perceptual dimensions are noisiness, coloration, discontinuity, and loudness. We create a crowdsourcing implementation of a multi-dimensional subjective test following the scales from P.804 and extend it to include reverberation, the speech signal, and overall quality. We show the tool is both accurate and reproducible. The tool has been used in the ICASSP 2023 Speech Signal Improvement challenge and we show the utility of these speech quality dimensions in this challenge. The tool will be publicly available as open-source at https://github.com/microsoft/P.808.