Quantifying the Role of Textual Predictability in Automatic Speech Recognition

TL;DR

This paper introduces a novel method to quantify how much a speech recognition model relies on textual context, revealing insights into model performance and language-specific challenges.

Contribution

It proposes a new approach to measure the influence of textual predictability on ASR errors, enabling better diagnosis and understanding of model behavior.

Findings

Wav2Vec 2.0 models utilize textual context more than hybrid models.

Standard ASR systems perform poorly on African-American English due to acoustic-phonetic modeling failures.

The method helps diagnose and improve ASR systems.

Abstract

A long-standing question in automatic speech recognition research is how to attribute errors to the ability of a model to model the acoustics, versus its ability to leverage higher-order context (lexicon, morphology, syntax, semantics). We validate a novel approach which models error rates as a function of relative textual predictability, and yields a single number, $k$, which measures the effect of textual predictability on the recognizer. We use this method to demonstrate that a Wav2Vec 2.0-based model makes greater stronger use of textual context than a hybrid ASR model, in spite of not using an explicit language model, and also use it to shed light on recent results demonstrating poor performance of standard ASR systems on African-American English. We demonstrate that these mostly represent failures of acoustic--phonetic modelling. We show how this approach can be used straightforwardly in diagnosing and improving ASR.