Can I introduce my boyfriend to my grandmother? Evaluating Large Language Models Capabilities on Iranian Social Norm Classification

TL;DR

This paper introduces the Iranian Social Norms (ISN) dataset and evaluates large language models' ability to classify Iranian social norms, revealing significant cultural and linguistic challenges in AI understanding of social context.

Contribution

The paper presents the first Farsi social norms dataset and provides an empirical evaluation of LLMs on culturally specific norms, highlighting the impact of language and geography.

Findings

LLMs perform poorly on Iranian norms, especially in Farsi.

English prompts with geographic context improve LLM performance.

Culturally tailored datasets are crucial for accurate social norm classification.

Abstract

Creating globally inclusive AI systems demands datasets reflecting diverse social norms. Iran, with its unique cultural blend, offers an ideal case study, with Farsi adding linguistic complexity. In this work, we introduce the Iranian Social Norms (ISN) dataset, a novel collection of 1,699 Iranian social norms, including environments, demographic features, and scope annotation, alongside English translations. Our evaluation of 6 Large Language Models (LLMs) in classifying Iranian social norms, using a variety of prompts, uncovered critical insights into the impact of geographic and linguistic context. Results revealed a substantial performance gap in LLMs' comprehension of Iranian norms. Notably, while the geographic context in English prompts enhanced the performance, this effect was absent in Farsi, pointing to nuanced linguistic challenges. Particularly, performance was significantly worse for Iran-specific norms, emphasizing the importance of culturally tailored datasets. As the first Farsi dataset for social norm classification, ISN will facilitate crucial cross-cultural analyses, shedding light on how values differ across contexts and cultures.