Benchmarking Automatic Speech Recognition Models for African Languages

TL;DR

This study systematically benchmarks four leading ASR models across 13 African languages, revealing how model performance varies with data size and providing practical insights for developing ASR systems in low-resource language contexts.

Contribution

It offers a comprehensive comparison of state-of-the-art ASR models in African languages, analyzing their data efficiency, scalability, and decoding strategies in low-resource settings.

Findings

MMS and W2v-BERT are more data-efficient in very low-resource regimes.

XLS-R scales more effectively with increasing data.

Whisper performs best in mid-resource conditions.

Abstract

Automatic speech recognition (ASR) for African languages remains constrained by limited labeled data and the lack of systematic guidance on model selection, data scaling, and decoding strategies. Large pre-trained systems such as Whisper, XLS-R, MMS, and W2v-BERT have expanded access to ASR technology, but their comparative behavior in African low-resource contexts has not been studied in a unified and systematic way. In this work, we benchmark four state-of-the-art ASR models across 13 African languages, fine-tuning them on progressively larger subsets of transcribed data ranging from 1 to 400 hours. Beyond reporting error rates, we provide new insights into why models behave differently under varying conditions. We show that MMS and W2v-BERT are more data efficient in very low-resource regimes, XLS-R scales more effectively as additional data becomes available, and Whisper demonstrates advantages in mid-resource conditions. We also analyze where external language model decoding yields improvements and identify cases where it plateaus or introduces additional errors, depending on the alignment between acoustic and text resources. By highlighting the interaction between pre-training coverage, model architecture, dataset domain, and resource availability, this study offers practical and insights into the design of ASR systems for underrepresented languages.