NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models

TL;DR

This paper introduces NormAd, a framework and benchmark for evaluating large language models' ability to judge social acceptability across diverse cultural norms, revealing current limitations in socio-cultural reasoning.

Contribution

We propose NormAd, a novel evaluation framework and benchmark to assess LLMs' cultural adaptability, highlighting their struggles with social norms across different cultures.

Findings

LLMs perform poorly in judging social acceptability across cultures.

Models are more accurate in recognizing acceptable vs. unacceptable situations.

Performance drops significantly with abstract norms and less cultural context.

Abstract

To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82\%) lags behind humans (> 95\%). In settings with abstract values and country information, model performance drops substantially (< 60\%), while human accuracy remains high (> 90\%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.