Speech Quality Assessment in Crowdsourcing: Comparison Category Rating Method

TL;DR

This paper evaluates the adaptation of the Comparison Category Rating (CCR) speech quality assessment method for crowdsourcing, demonstrating its reliability and validity compared to traditional laboratory tests.

Contribution

It introduces a crowdsourcing adaptation of the CCR speech quality assessment method, showing it is reproducible, reliable, and comparable to laboratory-based tests.

Findings

CCR results are highly reproducible in crowdsourcing.

Crowdsourcing CCR correlates well with laboratory tests.

The method is suitable for large-scale speech quality evaluation.

Abstract

Traditionally, Quality of Experience (QoE) for a communication system is evaluated through a subjective test. The most common test method for speech QoE is the Absolute Category Rating (ACR), in which participants listen to a set of stimuli, processed by the underlying test conditions, and rate their perceived quality for each stimulus on a specific scale. The Comparison Category Rating (CCR) is another standard approach in which participants listen to both reference and processed stimuli and rate their quality compared to the other one. The CCR method is particularly suitable for systems that improve the quality of input speech. This paper evaluates an adaptation of the CCR test procedure for assessing speech quality in the crowdsourcing set-up. The CCR method was introduced in the ITU-T Rec. P.800 for laboratory-based experiments. We adapted the test for the crowdsourcing approach following the guidelines from ITU-T Rec. P.800 and P.808. We show that the results of the CCR procedure via crowdsourcing are highly reproducible. We also compared the CCR test results with widely used ACR test procedures obtained in the laboratory and crowdsourcing. Our results show that the CCR procedure in crowdsourcing is a reliable and valid test method.