Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

TL;DR

This paper critiques the prevalent closed-choice survey methods for evaluating cultural alignment in LLMs, demonstrating that more flexible, unconstrained approaches reveal stronger alignment and expose limitations of traditional evaluations.

Contribution

It introduces unconstrained evaluation methods using real-world data, highlighting the variability and limitations of closed-style assessments for cultural alignment in LLMs.

Findings

LLMs show stronger cultural alignment in unconstrained settings

Minor survey changes cause inconsistent LLM responses

Closed-style evaluations have significant limitations

Abstract

A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.