Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation

TL;DR

QIMMA is a curated Arabic LLM evaluation benchmark that combines automated and human assessments to ensure high-quality, reliable Arabic NLP evaluation across multiple domains and tasks.

Contribution

It introduces a systematic, quality-first approach to Arabic benchmark validation, addressing issues in existing resources and providing a transparent, reproducible evaluation suite.

Findings

Curated over 52,000 samples grounded in native Arabic content.

Implemented a multi-model assessment pipeline with human review.

Released tools and data publicly for community use.

Abstract

We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation.