Classification errors distort findings in automated speech processing: examples and solutions from child-development research

TL;DR

This paper highlights how classification errors in automated speech analysis can distort research findings in child development studies and proposes a Bayesian method to measure and mitigate these effects.

Contribution

It introduces a Bayesian approach to assess and correct the downstream impact of classification errors on scientific inferences in speech processing research.

Findings

Classification errors significantly distort effect size estimates.

Bayesian calibration can partially recover unbiased estimates.

Errors impact commonly used speech classifiers like na and Voice Type Classifier.

Abstract

With the advent of wearable recorders, scientists are increasingly turning to automated methods of analysis of audio and video data in order to measure children's experience, behavior, and outcomes, with a sizable literature employing long-form audio-recordings to study language acquisition. While numerous articles report on the accuracy and reliability of the most popular automated classifiers, less has been written on the downstream effects of classification errors on measurements and statistical inferences (e.g., the estimate of correlations and effect sizes in regressions). This paper's main contributions are drawing attention to downstream effects of confusion errors, and providing an approach to measure and potentially recover from these errors. Specifically, we use a Bayesian approach to study the effects of algorithmic errors on key scientific questions, including the effect of siblings on children's language experience and the association between children's production and their input. By fitting a joint model of speech behavior and algorithm behavior on real and simulated data, we show that classification errors can significantly distort estimates for both the most commonly used \gls{lena}, and a slightly more accurate open-source alternative (the Voice Type Classifier from the ACLEW system). We further show that a Bayesian calibration approach for recovering unbiased estimates of effect sizes can be effective and insightful, but does not provide a fool-proof solution.