Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectations

TL;DR

This paper presents a human-centered framework for evaluating how well large language models reflect native cultural perceptions, revealing Western bias and systemic errors in cultural representation.

Contribution

It introduces a method to compare model-generated cultural vectors with human-derived importance vectors across nine countries, highlighting biases and systemic errors.

Findings

Models show Western-centric calibration, especially for non-US cultures.

High correlation of systemic errors across models indicates shared cultural neglect.

The approach advances beyond diversity metrics to assess cultural fidelity in AI outputs.

Abstract

Cultural representation in Large Language Model (LLM) outputs has primarily been evaluated through the proxies of cultural diversity and factual accuracy. However, a crucial gap remains in assessing cultural alignment: the degree to which generated content mirrors how native populations perceive and prioritize their own cultural facets. In this paper, we introduce a human-centered framework to evaluate the alignment of LLM generations with local expectations. First, we establish a human-derived ground-truth baseline of importance vectors, called Cultural Importance Vectors based on an induced set of culturally significant facets from open-ended survey responses collected across nine countries. Next, we introduce a method to compute model-derived Cultural Representation Vectors of an LLM based on a syntactically diversified prompt-set and apply it to three frontier LLMs (Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Haiku). Our investigation of the alignment between the human-derived Cultural Importance and model-derived Cultural Representations reveals a Western-centric calibration for some of the models where alignment decreases as a country's cultural distance from the US increases. Furthermore, we identify highly correlated, systemic error signatures ($\rho > 0.97$) across all models, which over-index on some cultural markers while neglecting the deep-seated social and value-based priorities of users. Our approach moves beyond simple diversity metrics toward evaluating the fidelity of AI-generated content in authentically capturing the nuanced hierarchies of global cultures.