Evaluating ASR robustness to spontaneous speech errors: A study of WhisperX using a Speech Error Database

TL;DR

This study uses the SFUSED database to evaluate WhisperX’s robustness to spontaneous speech errors, providing insights into its performance on linguistically annotated error data.

Contribution

It introduces a novel evaluation approach for ASR systems using detailed speech error annotations from SFUSED, highlighting WhisperX’s strengths and weaknesses.

Findings

WhisperX's accuracy varies across different error types.

The database effectively diagnoses ASR system performance.

Speech errors impact recognition accuracy significantly.

Abstract

The Simon Fraser University Speech Error Database (SFUSED) is a public data collection developed for linguistic and psycholinguistic research. Here we demonstrate how its design and annotations can be used to test and evaluate speech recognition models. The database comprises systematically annotated speech errors from spontaneous English speech, with each error tagged for intended and actual error productions. The annotation schema incorporates multiple classificatory dimensions that are of some value to model assessment, including linguistic hierarchical level, contextual sensitivity, degraded words, word corrections, and both word-level and syllable-level error positioning. To assess the value of these classificatory variables, we evaluated the transcription accuracy of WhisperX across 5,300 documented word and phonological errors. This analysis demonstrates the atabase's effectiveness as a diagnostic tool for ASR system performance.