Can We Trust Machine Learning? The Reliability of Features from Open-Source Speech Analysis Tools for Speech Modeling

TL;DR

This paper evaluates the reliability of open-source speech analysis tools, OpenSMILE and Praat, in extracting features for machine learning models applied to adolescents with autism, highlighting variability and potential biases.

Contribution

It provides an empirical assessment of the reliability of popular speech features across different tools and populations, emphasizing the need for domain-specific validation.

Findings

Significant variation in features across tools affects model performance.

Tool reliability varies across demographic groups and contexts.

Verification is essential for clinical and fair machine learning applications.

Abstract

Machine learning-based behavioral models rely on features extracted from audio-visual recordings. The recordings are processed using open-source tools to extract speech features for classification models. These tools often lack validation to ensure reliability in capturing behaviorally relevant information. This gap raises concerns about reproducibility and fairness across diverse populations and contexts. Speech processing tools, when used outside of their design context, can fail to capture behavioral variations equitably and can then contribute to bias. We evaluate speech features extracted from two widely used speech analysis tools, OpenSMILE and Praat, to assess their reliability when considering adolescents with autism. We observed considerable variation in features across tools, which influenced model performance across context and demographic groups. We encourage domain-relevant verification to enhance the reliability of machine learning models in clinical applications.