Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs

TL;DR

This paper introduces 'Fann or Flop', a comprehensive benchmark to evaluate large language models' understanding of Arabic poetry across historical eras and genres, highlighting current models' limitations in deep poetic comprehension.

Contribution

The paper presents the first extensive benchmark for assessing LLMs' grasp of Arabic poetry, covering multiple eras, genres, and poetic forms, with an open-source evaluation suite.

Findings

Most LLMs struggle with poetic understanding despite good performance on standard benchmarks.

Poetic comprehension requires deeper interpretive reasoning and cultural sensitivity.

The benchmark reveals significant gaps in current models' ability to understand Arabic poetry.

Abstract

Arabic poetry is one of the richest and most culturally rooted forms of expression in the Arabic language, known for its layered meanings, stylistic diversity, and deep historical continuity. Although large language models (LLMs) have demonstrated strong performance across languages and tasks, their ability to understand Arabic poetry remains largely unexplored. In this work, we introduce \emph{Fann or Flop}, the first benchmark designed to assess the comprehension of Arabic poetry by LLMs in 12 historical eras, covering 14 core poetic genres and a variety of metrical forms, from classical structures to contemporary free verse. The benchmark comprises a curated corpus of poems with explanations that assess semantic understanding, metaphor interpretation, prosodic awareness, and cultural context. We argue that poetic comprehension offers a strong indicator for testing how good the LLM understands classical Arabic through Arabic poetry. Unlike surface-level tasks, this domain demands deeper interpretive reasoning and cultural sensitivity. Our evaluation of state-of-the-art LLMs shows that most models struggle with poetic understanding despite strong results on standard Arabic benchmarks. We release "Fann or Flop" along with the evaluation suite as an open-source resource to enable rigorous evaluation and advancement for Arabic language models. Code is available at: https://github.com/mbzuai-oryx/FannOrFlop.