Are ASR foundation models generalized enough to capture features of regional dialects for low-resource languages?

TL;DR

This paper evaluates the ability of speech foundation models to recognize regional dialects in low-resource languages, revealing significant challenges and proposing dialect-specific training as a solution.

Contribution

It introduces a new Bengali dialect speech dataset and demonstrates the limitations of current models in dialectal ASR, highlighting the need for dialect-specific approaches.

Findings

Speech foundation models perform poorly on dialectal ASR tasks.

Dialect-specific training improves recognition accuracy.

The dataset serves as an out-of-distribution benchmark for low-resource ASR.

Abstract

Conventional research on speech recognition modeling relies on the canonical form for most low-resource languages while automatic speech recognition (ASR) for regional dialects is treated as a fine-tuning task. To investigate the effects of dialectal variations on ASR we develop a 78-hour annotated Bengali Speech-to-Text (STT) corpus named Ben-10. Investigation from linguistic and data-driven perspectives shows that speech foundation models struggle heavily in regional dialect ASR, both in zero-shot and fine-tuned settings. We observe that all deep learning methods struggle to model speech data under dialectal variations but dialect specific model training alleviates the issue. Our dataset also serves as a out of-distribution (OOD) resource for ASR modeling under constrained resources in ASR algorithms. The dataset and code developed for this project are publicly available