CAMEL-Bench: A Comprehensive Arabic LMM Benchmark

TL;DR

CAMEL-Bench is a comprehensive Arabic multimodal model benchmark covering diverse domains, designed to evaluate and improve LMMs for Arabic visual reasoning tasks, with open-source tools for community use.

Contribution

This work introduces the first large-scale, multi-domain Arabic LMM benchmark with manually verified questions, filling a significant gap in multilingual multimodal evaluation.

Findings

GPT-4o scored 62% overall, indicating room for improvement.

Open-source models lag behind closed-source counterparts.

Benchmark covers 8 domains with 29,036 questions.

Abstract

Recent years have witnessed a significant interest in developing large multimodal models (LMMs) capable of performing various visual reasoning and understanding tasks. This has led to the introduction of multiple LMM benchmarks to evaluate LMMs on different tasks. However, most existing LMM evaluation benchmarks are predominantly English-centric. In this work, we develop a comprehensive LMM evaluation benchmark for the Arabic language to represent a large population of over 400 million speakers. The proposed benchmark, named CAMEL-Bench, comprises eight diverse domains and 38 sub-domains including, multi-image understanding, complex visual perception, handwritten document understanding, video understanding, medical imaging, plant diseases, and remote sensing-based land use understanding to evaluate broad scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions that are filtered from a larger pool of samples, where the quality is manually verified by native speakers to ensure reliable model assessment. We conduct evaluations of both closed-source, including GPT-4 series, and open-source LMMs. Our analysis reveals the need for substantial improvement, especially among the best open-source models, with even the closed-source GPT-4o achieving an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.