MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
Ghazal Kalhor, Behnam Bahrak

TL;DR
MasalBench is a new benchmark designed to evaluate how well multilingual LLMs understand Persian proverbs in context and across cultures, revealing strengths in proverb recognition but limitations in cultural and analogical reasoning.
Contribution
Introduces MasalBench, the first comprehensive benchmark for assessing LLMs' understanding of Persian proverbs and cross-cultural analogies in a low-resource language.
Findings
LLMs perform above 0.90 in identifying Persian proverbs in context
Performance drops to 0.79 in identifying English equivalents of Persian proverbs
Current LLMs show limitations in cultural knowledge and analogical reasoning
Abstract
In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsLanguage, Metaphor, and Cognition · Language and cultural evolution · Natural Language Processing Techniques
