Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces

TL;DR

This paper evaluates the digital presence of African languages and finds that curated news data improves AI language detection accuracy more than conversational data, highlighting challenges in processing code-switched and sparse online language use.

Contribution

It demonstrates the effectiveness of using curated news data over social media for training language detection models on African languages and emphasizes the need for models handling code-switching.

Findings

News media provides more reliable data than social media for African languages.

Language detection models perform well on clean news data but struggle with code-switched social media posts.

Future models should better handle code-switching and sparse conversational data.

Abstract

This study examines the digital representation of African languages and the challenges this presents for current language detection tools. We evaluate their performance on Yoruba, Kinyarwanda, and Amharic. While these languages are spoken by millions, their online usage on conversational platforms is often sparse, heavily influenced by English, and not representative of the authentic, monolingual conversations prevalent among native speakers. This lack of readily available authentic data online creates a challenge of scarcity of conversational data for training language models. To investigate this, data was collected from subreddits and local news sources for each language. The analysis showed a stark contrast between the two sources. Reddit data was minimal and characterized by heavy code-switching. Conversely, local news media offered a robust source of clean, monolingual language data, which also prompted more user engagement in the local language on the news publishers social media pages. Language detection models, including the specialized AfroLID and a general LLM, performed with near-perfect accuracy on the clean news data but struggled with the code-switched Reddit posts. The study concludes that professionally curated news content is a more reliable and effective source for training context-rich AI models for African languages than data from conversational platforms. It also highlights the need for future models that can process clean and code-switched text to improve the detection accuracy for African languages.