Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants

TL;DR

This paper introduces a new Arabic cultural QA benchmark with dialect variants, translating questions into multiple dialects, converting them into open-ended formats, and evaluating LLMs' performance with chain-of-thought reasoning.

Contribution

It presents the first parallel dataset of Arabic QA across dialects, extending existing datasets, and evaluates LLMs' performance on culturally grounded, dialect-specific questions.

Findings

Models underperform on Arabic dialects, especially on open-ended questions.

Arabic-centric models excel at MCQs but struggle with OEQs.

Chain-of-thought reasoning improves correctness judgments but affects n-gram metrics.

Abstract

Large Language Models (LLMs) are increasingly used to answer everyday questions, yet their performance on culturally grounded and dialectal content remains uneven across languages. We propose a comprehensive method that (i) translates Modern Standard Arabic (MSA) multiple-choice questions (MCQs) into English and several Arabic dialects, (ii) converts them into open-ended questions (OEQs), (iii) benchmarks a range of zero-shot and fine-tuned LLMs under both MCQ and OEQ settings, and (iv) generates chain-of-thought (CoT) rationales to fine-tune models for step-by-step reasoning. Using this method, we extend an existing dataset in which QAs are parallelly aligned across multiple language varieties, making it, to our knowledge, the first of its kind. We conduct extensive experiments with both open and closed models. Our findings show that (i) models underperform on Arabic dialects, revealing persistent gaps in culturally grounded and dialect-specific knowledge; (ii) Arabic-centric models perform well on MCQs but struggle with OEQs; and (iii) CoT improves judged correctness while yielding mixed n-gram-based metrics. The developed dataset will be publicly released to support further research on culturally and linguistically inclusive evaluation.