GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek

TL;DR

GreekMMLU is a comprehensive, native-sourced Greek language benchmark with 21,805 questions across diverse subjects, designed to evaluate and improve multilingual language models' performance in Greek.

Contribution

It introduces the first large-scale, authentic Greek language understanding benchmark with detailed taxonomy and difficulty levels, enabling robust evaluation of LLMs in Greek.

Findings

Open- and closed-source LLMs show significant performance gaps.

Greek-adapted models outperform general multilingual models.

Model scale, adaptation, and prompting significantly influence performance.

Abstract

Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek.