A Toolkit for Detecting Spurious Correlations in Speech Datasets

TL;DR

This paper presents a toolkit designed to detect spurious correlations in speech datasets, especially those arising from recording conditions, which can lead to misleading performance estimates in high-stakes applications.

Contribution

The authors introduce a diagnostic toolkit that identifies spurious correlations by analyzing non-speech regions, improving reliability in speech dataset evaluation.

Findings

Toolkit effectively detects spurious correlations in speech datasets.

Better-than-chance performance on non-speech regions indicates potential spurious cues.

Publicly available toolkit facilitates more robust speech dataset analysis.

Abstract

We introduce a toolkit for uncovering spurious correlations between recording characteristics and target class in speech datasets. Spurious correlations may arise due to heterogeneous recording conditions, a common scenario for health-related datasets. When present both in the training and test data, these correlations result in an overestimation of the system performance -- a dangerous situation, specially in high-stakes application where systems are required to satisfy minimum performance requirements. Our toolkit implements a diagnostic method based on the detection of the target class using only the non-speech regions in the audio. Better than chance performance at this task indicates that information about the target class can be extracted from the non-speech regions, flagging the presence of spurious correlations. The toolkit is publicly available for research use.