Comparison of remote experiments using crowdsourcing and laboratory experiments on speech intelligibility

TL;DR

This study compares speech intelligibility results from remote crowdsourcing experiments and traditional laboratory tests, finding that despite higher variability in remote results, they can still be valuable for developing objective speech measures.

Contribution

It demonstrates that remote crowdsourcing experiments yield comparable variance patterns to laboratory tests, supporting their use in speech intelligibility research during constraints like a pandemic.

Findings

Remote experiments have higher mean and SD of SRT than lab tests.

Variance patterns across conditions are similar in both methods.

Practice scores correlate with SRT, aiding data screening.

Abstract

Many subjective experiments have been performed to develop objective speech intelligibility measures, but the novel coronavirus outbreak has made it very difficult to conduct experiments in a laboratory. One solution is to perform remote testing using crowdsourcing; however, because we cannot control the listening conditions, it is unclear whether the results are entirely reliable. In this study, we compared speech intelligibility scores obtained in remote and laboratory experiments. The results showed that the mean and standard deviation (SD) of the remote experiments' speech reception threshold (SRT) were higher than those of the laboratory experiments. However, the variance in the SRTs across the speech-enhancement conditions revealed similarities, implying that remote testing results may be as useful as laboratory experiments to develop an objective measure. We also show that the practice session scores correlate with the SRT values. This is a priori information before performing the main tests and would be useful for data screening to reduce the variability of the SRT distribution.