Effective data screening technique for crowdsourced speech intelligibility experiments: Evaluation with IRM-based speech enhancement

TL;DR

This paper presents a simple tone pip test method for screening data in crowdsourced speech intelligibility experiments, improving reliability and enabling comparison with laboratory results, especially for speech enhanced by ideal ratio masks.

Contribution

Introduces a tone pip-based data screening technique for crowdsourced speech intelligibility tests, ensuring data quality comparable to controlled laboratory experiments.

Findings

Tone pip tests effectively filter unreliable data

Crowdsourced results align closely with laboratory data after screening

Upper limit of speech enhancement performance established with IRM

Abstract

It is essential to perform speech intelligibility (SI) experiments with human listeners in order to evaluate objective intelligibility measures for developing effective speech enhancement and noise reduction algorithms. Recently, crowdsourced remote testing has become a popular means for collecting a massive amount and variety of data at a relatively small cost and in a short time. However, careful data screening is essential for attaining reliable SI data. We performed SI experiments on speech enhanced by an "oracle" ideal ratio mask (IRM) in a well-controlled laboratory and in crowdsourced remote environments that could not be controlled directly. We introduced simple tone pip tests, in which participants were asked to report the number of audible tone pips, to estimate their listening levels above audible thresholds. The tone pip tests were very effective for data screening to reduce the variability of crowdsourced remote results so that the laboratory results would become similar. The results also demonstrated the SI of an oracle IRM, giving us the upper limit of the mask-based single-channel speech enhancement.