Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks

TL;DR

This paper benchmarks large language models' ability to understand and reason in Arabic medical tasks, revealing their strengths and limitations in clinical question answering and semantic alignment.

Contribution

It introduces a comprehensive evaluation of LLMs on Arabic medical NLP tasks using the AraHealthQA dataset, highlighting effective models and novel voting strategies.

Findings

Gemini and GPT models achieved up to 77% accuracy on MCQs.

Several LLMs reached a BERTScore of 86.44% on open-ended questions.

Majority voting improved overall performance in medical question answering.

Abstract

Recent progress in large language models (LLMs) has showcased impressive proficiency in numerous Arabic natural language processing (NLP) applications. Nevertheless, their effectiveness in Arabic medical NLP domains has received limited investigation. This research examines the degree to which state-of-the-art LLMs demonstrate and articulate healthcare knowledge in Arabic, assessing their capabilities across a varied array of Arabic medical tasks. We benchmark several LLMs using a medical dataset proposed in the Arabic NLP AraHealthQA challenge in MedArabiQ2025 track. Various base LLMs were assessed on their ability to accurately provide correct answers from existing choices in multiple-choice questions (MCQs) and fill-in-the-blank scenarios. Additionally, we evaluated the capacity of LLMs in answering open-ended questions aligned with expert answers. Our results reveal significant variations in correct answer prediction accuracy and low variations in semantic alignment of generated answers, highlighting both the potential and limitations of current LLMs in Arabic clinical contexts. Our analysis shows that for MCQs task, the proposed majority voting solution, leveraging three base models (Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3), outperforms others, achieving up to 77% accuracy and securing first place overall in the Arahealthqa 2025 shared task-track 2 (sub-task 1) challenge. Moreover, for the open-ended questions task, several LLMs were able to demonstrate excellent performance in terms of semantic alignment and achieve a maximum BERTScore of 86.44%.