AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic

TL;DR

AmharicStoryQA is a culturally diverse benchmark for evaluating Amharic language models' narrative understanding, revealing regional differences and highlighting the importance of culturally grounded assessments in low-resource languages.

Contribution

This work introduces AmharicStoryQA, a novel culturally diverse story question answering benchmark for Amharic, addressing the gap in evaluating regional and cultural understanding in language models.

Findings

Existing LLMs show significant narrative understanding gaps.

Regional differences significantly affect evaluation outcomes.

Supervised fine-tuning yields uneven improvements across regions.

Abstract

With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages.