MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks

TL;DR

This paper introduces MedArabiQ, a comprehensive benchmark dataset for evaluating large language models on Arabic medical tasks, addressing a critical gap in multilingual healthcare AI research.

Contribution

The study presents the first Arabic medical benchmark dataset with diverse tasks, enabling systematic evaluation of LLMs in Arabic healthcare applications.

Findings

LLMs show varying performance on Arabic medical tasks.

Highlight the need for high-quality multilingual benchmarks.

Demonstrated the importance of bias mitigation in medical LLMs.

Abstract

Large Language Models (LLMs) have demonstrated significant promise for various applications in healthcare. However, their efficacy in the Arabic medical domain remains unexplored due to the lack of high-quality domain-specific datasets and benchmarks. This study introduces MedArabiQ, a novel benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and including multiple choice questions, fill-in-the-blank, and patient-doctor question answering. We first constructed the dataset using past medical exams and publicly available datasets. We then introduced different modifications to evaluate various LLM capabilities, including bias mitigation. We conducted an extensive evaluation with five state-of-the-art open-source and proprietary LLMs, including GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5. Our findings highlight the need for the creation of new high-quality benchmarks that span different languages to ensure fair deployment and scalability of LLMs in healthcare. By establishing this benchmark and releasing the dataset, we provide a foundation for future research aimed at evaluating and enhancing the multilingual capabilities of LLMs for the equitable use of generative AI in healthcare.