English Accent Accuracy Analysis in a State-of-the-Art Automatic Speech Recognition System

TL;DR

This study evaluates a deep learning-based ASR system's performance across diverse English accents, revealing biases favoring prevalent accents and highlighting the impact of accent diversity on recognition accuracy.

Contribution

The paper provides a comprehensive analysis of accent-related biases in a state-of-the-art ASR system using diverse, unseen English accent data from multiple sources.

Findings

Bias towards prevalent accents in training data

Recognition accuracy varies significantly across accents

Accent diversity impacts overall ASR performance

Abstract

Nowadays, research in speech technologies has gotten a lot out thanks to recently created public domain corpora that contain thousands of recording hours. These large amounts of data are very helpful for training the new complex models based on deep learning technologies. However, the lack of dialectal diversity in a corpus is known to cause performance biases in speech systems, mainly for underrepresented dialects. In this work, we propose to evaluate a state-of-the-art automatic speech recognition (ASR) deep learning-based model, using unseen data from a corpus with a wide variety of labeled English accents from different countries around the world. The model has been trained with 44.5K hours of English speech from an open access corpus called Multilingual LibriSpeech, showing remarkable results in popular benchmarks. We test the accuracy of such ASR against samples extracted from another public corpus that is continuously growing, the Common Voice dataset. Then, we present graphically the accuracy in terms of Word Error Rate of each of the different English included accents, showing that there is indeed an accuracy bias in terms of accentual variety, favoring the accents most prevalent in the training corpus.