Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach

TL;DR

This paper introduces a trait-centric prompt engineering framework for Arabic essay scoring using large language models, demonstrating that structured prompting improves assessment accuracy across proficiency traits in low-resource settings.

Contribution

It proposes a novel three-tier prompting strategy for Arabic AES, including hybrid and rubric-guided methods, and evaluates their effectiveness with multiple LLMs on a new Arabic dataset.

Findings

Fanar-1-9B-Instruct achieved highest trait agreement (QWK=0.28)

Rubric-guided prompting improved trait evaluation consistency

Structured prompting enhances Arabic AES effectiveness regardless of model size

Abstract

This paper presents a novel prompt engineering framework for trait specific Automatic Essay Scoring (AES) in Arabic, leveraging large language models (LLMs) under zero-shot and few-shot configurations. Addressing the scarcity of scalable, linguistically informed AES tools for Arabic, we introduce a three-tier prompting strategy (standard, hybrid, and rubric-guided) that guides LLMs in evaluating distinct language proficiency traits such as organization, vocabulary, development, and style. The hybrid approach simulates multi-agent evaluation with trait specialist raters, while the rubric-guided method incorporates scored exemplars to enhance model alignment. In zero and few-shot settings, we evaluate eight LLMs on the QAES dataset, the first publicly available Arabic AES resource with trait level annotations. Experimental results using Quadratic Weighted Kappa (QWK) and Confidence Intervals show that Fanar-1-9B-Instruct achieves the highest trait level agreement in both zero and few-shot prompting (QWK = 0.28 and CI = 0.41), with rubric-guided prompting yielding consistent gains across all traits and models. Discourse-level traits such as Development and Style showed the greatest improvements. These findings confirm that structured prompting, not model scale alone, enables effective AES in Arabic. Our study presents the first comprehensive framework for proficiency oriented Arabic AES and sets the foundation for scalable assessment in low resource educational contexts.