MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment

TL;DR

MEENA (PersianMMMU) is a comprehensive dataset designed to evaluate Persian vision-language models across diverse educational and cultural tasks, addressing the gap in multilingual model assessment.

Contribution

Introduces MEENA, the first Persian VLM benchmark with diverse questions, metadata, and bilingual structure to evaluate cross-linguistic and multimodal understanding.

Findings

Assesses model performance across multiple tasks and subjects.

Highlights challenges in Persian multimodal understanding.

Provides insights into model hallucinations and attention capabilities.

Abstract

Recent advancements in large vision-language models (VLMs) have primarily focused on English, with limited attention given to other languages. To address this gap, we introduce MEENA (also known as PersianMMMU), the first dataset designed to evaluate Persian VLMs across scientific, reasoning, and human-level understanding tasks. Our dataset comprises approximately 7,500 Persian and 3,000 English questions, covering a wide range of topics such as reasoning, mathematics, physics, diagrams, charts, and Persian art and literature. Key features of MEENA include: (1) diverse subject coverage spanning various educational levels, from primary to upper secondary school, (2) rich metadata, including difficulty levels and descriptive answers, (3) original Persian data that preserves cultural nuances, (4) a bilingual structure to assess cross-linguistic performance, and (5) a series of diverse experiments assessing various capabilities, including overall performance, the model's ability to attend to images, and its tendency to generate hallucinations. We hope this benchmark contributes to enhancing VLM capabilities beyond English.