Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

TL;DR

The Khayyam Challenge (PersianMMLU) provides a comprehensive, culturally nuanced benchmark with over 20,000 questions to evaluate Persian-supporting LLMs across diverse subjects and educational levels.

Contribution

It introduces a new, extensive Persian language evaluation dataset with rich metadata, avoiding translation issues, and offers a scalable framework for assessing LLMs' language understanding and reasoning.

Findings

Existing LLMs show varied performance across tasks.

The benchmark reveals strengths and weaknesses of Persian LLMs.

Rich metadata enables detailed analysis of model capabilities.

Abstract

Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages. In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc., aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs.