GhazalBench: Usage-Grounded Evaluation of LLMs on Persian Ghazals

TL;DR

GhazalBench is a new benchmark designed to evaluate large language models' ability to understand, paraphrase, and recall Persian ghazals, revealing strengths in meaning comprehension but challenges in exact verse recall, especially compared to English sonnets.

Contribution

Introduces GhazalBench, a comprehensive evaluation framework for LLMs on Persian ghazals, emphasizing culturally grounded understanding and form-based recall.

Findings

Models understand poetic meaning well.

Models struggle with exact verse recall.

Recognition tasks improve recall performance.

Abstract

Persian poetry plays an active role in Iranian cultural practice, where verses by canonical poets such as Hafez are frequently quoted, paraphrased, or completed from partial cues. Supporting such interactions requires language models to engage not only with poetic meaning but also with culturally entrenched surface form. We introduce GhazalBench, a benchmark for evaluating how large language models (LLMs) interact with Persian ghazals under usage-grounded conditions. GhazalBench assesses two complementary abilities: producing faithful prose paraphrases of couplets and accessing canonical verses under varying semantic and formal cues. Across several proprietary and open-weight multilingual LLMs, we observe a consistent dissociation: models generally capture poetic meaning but struggle with exact verse recall in completion-based settings, while recognition-based tasks substantially reduce this gap. A parallel evaluation on English sonnets shows markedly higher recall performance, suggesting that these limitations are tied to differences in training exposure rather than inherent architectural constraints. Our findings highlight the need for evaluation frameworks that jointly assess meaning, form, and cue-dependent access to culturally significant texts. GhazalBench is available at https://github.com/kalhorghazal/GhazalBench.