MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark

TL;DR

MedAraBench is a comprehensive Arabic medical question-answering dataset that enables evaluation and development of multilingual LLMs in healthcare, addressing a critical resource gap for Arabic NLP in medicine.

Contribution

The paper introduces MedAraBench, a large-scale, high-quality Arabic medical dataset with diverse specialties, and provides benchmark evaluations of current models to advance Arabic medical NLP research.

Findings

Existing models show room for domain-specific improvements.

The dataset covers 19 specialties and five difficulty levels.

Benchmark results highlight the need for further model enhancements.

Abstract

Arabic remains one of the most underrepresented languages in natural language processing research, particularly in medical applications, due to the limited availability of open-source data and benchmarks. The lack of resources hinders efforts to evaluate and advance the multilingual capabilities of Large Language Models (LLMs). In this paper, we introduce MedAraBench, a large-scale dataset consisting of Arabic multiple-choice question-answer pairs across various medical specialties. We constructed the dataset by manually digitizing a large repository of academic materials created by medical professionals in the Arabic-speaking region. We then conducted extensive preprocessing and split the dataset into training and test sets to support future research efforts in the area. To assess the quality of the data, we adopted two frameworks, namely expert human evaluation and LLM-as-a-judge. Our dataset is diverse and of high quality, spanning 19 specialties and five difficulty levels. For benchmarking purposes, we assessed the performance of eight state-of-the-art open-source and proprietary models, such as GPT-5, Gemini 2.0 Flash, and Claude 4-Sonnet. Our findings highlight the need for further domain-specific enhancements. We release the dataset and evaluation scripts to broaden the diversity of medical data benchmarks, expand the scope of evaluation suites for LLMs, and enhance the multilingual capabilities of models for deployment in clinical settings.