AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic

TL;DR

This paper introduces a comprehensive framework for evaluating large language models' performance in dialectal Arabic, revealing biases and challenges in generating and understanding this underrepresented language variety.

Contribution

It provides an operationalized assessment framework for LLMs in dialectal Arabic and offers practical insights into their strengths and limitations.

Findings

LLMs understand dialectal Arabic better than they generate it.

Post-training can introduce bias against dialectal Arabic.

Few-shot examples help improve LLM performance in dialectal Arabic.

Abstract

Dialectal Arabic (DA) varieties are under-served by language technologies, particularly large language models (LLMs). This trend threatens to exacerbate existing social inequalities and limits LLM applications, yet the research community lacks operationalized performance measurements in DA. We present a framework that comprehensively assesses LLMs' DA modeling capabilities across four dimensions: fidelity, understanding, quality, and diglossia. We evaluate nine LLMs in eight DA varieties and provide practical recommendations. Our evaluation suggests that LLMs do not produce DA as well as they understand it, not because their DA fluency is poor, but because they are reluctant to generate DA. Further analysis suggests that current post-training can contribute to bias against DA, that few-shot examples can overcome this deficiency, and that otherwise no measurable features of input text correlate well with LLM DA performance.