Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance

TL;DR

This study investigates racial disparities in ASR performance, revealing that differences in recording practices and dialectal variation significantly affect accuracy, emphasizing the need for systematic analysis of bias sources.

Contribution

It identifies the impact of recording provenance and dialectal variation on ASR accuracy, highlighting confounding factors previously overlooked in fairness assessments.

Findings

Significant dialectal variation affects ASR performance.

Fine-tuning improves accuracy on African American English.

Recording quality differences confound performance assessments.

Abstract

Automatic speech recognition (ASR) models trained on large amounts of audio data are now widely used to convert speech to written text in a variety of applications from video captioning to automated assistants used in healthcare and other domains. As such, it is important that ASR models and their use is fair and equitable. Prior work examining the performance of commercial ASR systems on the Corpus of Regional African American Language (CORAAL) demonstrated significantly worse ASR performance on African American English (AAE). The current study seeks to understand the factors underlying this disparity by examining the performance of the current state-of-the-art neural network based ASR system (Whisper, OpenAI) on the CORAAL dataset. Two key findings have been identified as a result of the current study. The first confirms prior findings of significant dialectal variation even across neighboring communities, and worse ASR performance on AAE that can be improved to some extent with fine-tuning of ASR models. The second is a novel finding not discussed in prior work on CORAAL: differences in audio recording practices within the dataset have a significant impact on ASR accuracy resulting in a ``confounding by provenance'' effect in which both language use and recording quality differ by study location. These findings highlight the need for further systematic investigation to disentangle the effects of recording quality and inherent linguistic diversity when examining the fairness and bias present in neural ASR models, as any bias in ASR accuracy may have negative downstream effects on disparities in various domains of life in which ASR technology is used.