BALSAM: A Platform for Benchmarking Arabic Large Language Models

TL;DR

BALSAM is a comprehensive, community-driven benchmarking platform designed to evaluate and advance Arabic Large Language Models across diverse NLP tasks with transparent, blind testing.

Contribution

It introduces a large, diverse Arabic benchmark with 78 tasks and a centralized platform for unbiased evaluation, addressing previous limitations in Arabic NLP benchmarking.

Findings

Provides 78 NLP tasks covering broad categories

Includes 52K examples with blind test sets

Establishes a transparent platform for Arabic LLM evaluation

Abstract

The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.