Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

TL;DR

This paper introduces ArabCulture-Dialogue, a comprehensive dataset for evaluating cultural reasoning in Arabic language models across dialects and MSA, highlighting existing performance gaps.

Contribution

It presents a new culturally grounded conversational dataset covering 13 countries and three benchmarking tasks to assess dialectal and MSA Arabic understanding.

Findings

Models perform worse on dialects than MSA across tasks.

The dataset covers 12 topics and 54 subtopics.

Benchmarking reveals significant performance gaps.

Abstract

There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country's respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiple-choice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one.