CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs

TL;DR

This paper introduces CURE, a comprehensive framework for evaluating cultural understanding in large language models through realistic scenarios and multiple metrics, revealing limitations of current thin evaluations.

Contribution

It proposes a novel 'thick' evaluation framework with realistic contexts and multiple metrics, improving assessment of cultural reasoning in LLMs.

Findings

Thin evaluations overestimate cultural competence.

Thick evaluations reveal reasoning depth and reduce assessment variance.

Thick evaluation provides more stable and interpretable cultural understanding signals.

Abstract

Large language models (LLMs) are increasingly deployed in culturally diverse environments, yet existing evaluations of cultural competence remain limited. Existing methods focus on de-contextualized correctness or forced-choice judgments, overlooking the need for cultural understanding and reasoning required for appropriate responses. To address this gap, we introduce a set of benchmarks that, instead of directly probing abstract norms or isolated statements, present models with realistic situational contexts that require culturally grounded reasoning. In addition to the standard Exact Match metric, we introduce four complementary metrics (Coverage, Specificity, Connotation, and Coherence) to capture different dimensions of model's response quality. Empirical analysis across frontier models reveals that thin evaluation systematically overestimates cultural competence and produces unstable assessments with high variance. In contrast, thick evaluation exposes differences in reasoning depth, reduces variance, and provides more stable, interpretable signals of cultural understanding.