Afri-MCQA: Multimodal Cultural Question Answering for African Languages

TL;DR

This paper introduces Afri-MCQA, a comprehensive multilingual cultural question-answering benchmark for African languages, revealing significant performance gaps in current large language models and emphasizing the need for culturally and linguistically tailored AI approaches.

Contribution

The paper presents the first multilingual cultural QA benchmark for African languages, created entirely by native speakers, and evaluates LLMs, highlighting their poor performance and the need for culturally grounded models.

Findings

Open-weight LLMs perform poorly on African language QA tasks.

Significant performance gaps exist between native languages and English.

Speech-first approaches and culturally grounded pretraining are needed.

Abstract

Africa is home to over one-third of the world's languages, yet remains underrepresented in AI research. We introduce Afri-MCQA, the first Multilingual Cultural Question-Answering benchmark covering 7.5k Q&A pairs across 15 African languages from 12 countries. The benchmark offers parallel English-African language Q&A pairs across text and speech modalities and was entirely created by native speakers. Benchmarking large language models (LLMs) on Afri-MCQA shows that open-weight models perform poorly across evaluated cultures, with near-zero accuracy on open-ended VQA when queried in native language or speech. To evaluate linguistic competence, we include control experiments meant to assess this specific aspect separate from cultural knowledge, and we observe significant performance gaps between native languages and English for both text and speech. These findings underscore the need for speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer. To support more inclusive multimodal AI development in African languages, we release our Afri-MCQA under academic license or CC BY-NC 4.0 on HuggingFace (https://huggingface.co/datasets/Atnafu/Afri-MCQA)