TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

TL;DR

This paper introduces a Persian-specific short-answer benchmark for evaluating the cultural understanding of language models, utilizing a hybrid semantic similarity approach to improve scoring accuracy over traditional exact-match methods.

Contribution

It presents the first standardized Persian cultural evaluation benchmark with a hybrid scoring method that captures semantic nuance and morphological complexity.

Findings

Hybrid evaluation improves scoring consistency by +10 over exact-match baselines.

Semantic similarity metric aligns better with human judgments.

Framework is publicly released for future research and benchmarking.

Abstract

This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.