PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian

TL;DR

PerCul introduces a culturally nuanced dataset to evaluate Persian language models' sensitivity to Persian culture, revealing significant gaps in their cultural understanding compared to layperson benchmarks.

Contribution

This paper presents PerCul, a novel story-based dataset curated with native Persian input to assess LLMs' cultural competence in Persian, addressing a critical evaluation gap.

Findings

Best closed source model lags 11.3% behind layperson baseline.

Open-weight models show a 21.3% gap from layperson baseline.

PerCul provides a new benchmark for cross-cultural NLP evaluation.

Abstract

Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here: https://huggingface.co/datasets/teias-ai/percul