Where Are We At with Automatic Speech Recognition for the Bambara Language?

TL;DR

This paper establishes the first standardized benchmark for Bambara ASR, revealing current models' performance gaps and emphasizing the need for specialized approaches for underrepresented languages.

Contribution

It introduces a controlled benchmark dataset and evaluation framework for Bambara ASR, enabling consistent comparison and progress tracking in this underrepresented language.

Findings

Top WER achieved is 46.76%

Best CER achieved is 13.00%

Multilingual models often exceed 100% WER

Abstract

This paper introduces the first standardized benchmark for evaluating Automatic Speech Recognition (ASR) in the Bambara language, utilizing one hour of professionally recorded Malian constitutional text. Designed as a controlled reference set under near-optimal acoustic and linguistic conditions, the benchmark was used to evaluate 37 models, ranging from Bambara-trained systems to large-scale commercial models. Our findings reveal that current ASR performance remains significantly below deployment standards in a narrow formal domain; the top-performing system in terms of Word Error Rate (WER) achieved 46.76\% and the best Character Error Rate (CER) of 13.00\% was set by another model, while several prominent multilingual models exceeded 100\% WER. These results suggest that multilingual pre-training and model scaling alone are insufficient for underrepresented languages. Furthermore, because this dataset represents a best-case scenario of the most simplified and formal form of spoken Bambara, these figures are yet to be tested against practical, real-world settings. We provide the benchmark and an accompanying public leaderboard to facilitate transparent evaluation and future research in Bambara speech technology.