Robustness of end-to-end Automatic Speech Recognition Models -- A Case Study using Mozilla DeepSpeech

TL;DR

This paper investigates the robustness of end-to-end automatic speech recognition models, specifically Mozilla DeepSpeech, highlighting how dataset biases and overlaps can significantly underestimate true error rates.

Contribution

It provides a detailed analysis of how dataset selection bias, gender, and content overlap affect the perceived performance of speech recognition models.

Findings

Content overlap greatly impacts error rates

Gender influences model performance

Dataset biases can underestimate true error rates

Abstract

When evaluating the performance of automatic speech recognition models, usually word error rate within a certain dataset is used. Special care must be taken in understanding the dataset in order to report realistic performance numbers. We argue that many performance numbers reported probably underestimate the expected error rate. We conduct experiments controlling for selection bias, gender as well as overlap (between training and test data) in content, voices, and recording conditions. We find that content overlap has the biggest impact, but other factors like gender also play a role.