3LM: Bridging Arabic, STEM, and Code through Benchmarking

TL;DR

This paper introduces 3LM, a comprehensive benchmark suite for Arabic language models, covering STEM and code domains, to address the lack of evaluation resources in these critical areas.

Contribution

The paper presents the first dedicated Arabic benchmarks for STEM and code, including question-answer pairs, synthetic questions, and translated code benchmarks, to advance Arabic LLM research.

Findings

Benchmarks publicly released for community use

Supports evaluation of Arabic LLMs in STEM and coding

Addresses a significant gap in Arabic NLP resources

Abstract

Arabic is one of the most widely spoken languages in the world, yet efforts to develop and evaluate Large Language Models (LLMs) for Arabic remain relatively limited. Most existing Arabic benchmarks focus on linguistic, cultural, or religious content, leaving a significant gap in domains like STEM and code which are increasingly relevant for real-world LLM applications. To help bridge this gap, we present 3LM, a suite of three benchmarks designed specifically for Arabic. The first is a set of STEM-related question-answer pairs, naturally sourced from Arabic textbooks and educational worksheets. The second consists of synthetically generated STEM questions, created using the same sources. The third benchmark focuses on code generation, built through a careful translation of two widely used code benchmarks, incorporating a human-in-the-loop process with several rounds of review to ensure high-quality and faithful translations. We release all three benchmarks publicly to support the growth of Arabic LLM research in these essential but underrepresented areas.