Clustering and Mining Accented Speech for Inclusive and Fair Speech Recognition

TL;DR

This paper proposes accent clustering and mining techniques to improve fairness in speech recognition systems, especially for under-represented accents, by enhancing accent recognition and fine-tuning ASR models.

Contribution

It introduces novel accent clustering and mining schemes, including supervised/unsupervised pre-training, DRO, and clustering, to address accent data imbalance in speech recognition.

Findings

Improved accent recognition accuracy for unbalanced data.

10.0% relative improvement in ASR performance on Indian accent after clustering.

5.3% relative improvement in ASR performance with unsupervised clustering.

Abstract

Modern automatic speech recognition (ASR) systems are typically trained on more than tens of thousands hours of speech data, which is one of the main factors for their great success. However, the distribution of such data is typically biased towards common accents or typical speech patterns. As a result, those systems often poorly perform on atypical accented speech. In this paper, we present accent clustering and mining schemes for fair speech recognition systems which can perform equally well on under-represented accented speech. For accent recognition, we applied three schemes to overcome limited size of supervised accent data: supervised or unsupervised pre-training, distributionally robust optimization (DRO) and unsupervised clustering. Three schemes can significantly improve the accent recognition model especially for unbalanced and small accented speech. Fine-tuning ASR on the mined Indian accent speech using the proposed supervised or unsupervised clustering schemes showed 10.0% and 5.3% relative improvements compared to fine-tuning on the randomly sampled speech, respectively.