On The Conceptualization and Societal Impact of Cross-Cultural Bias

TL;DR

This paper reviews recent literature on cultural bias in NLP, emphasizing the importance of stakeholder engagement and proposing a framework for conceptualizing and evaluating societal impacts of cross-cultural bias in language models.

Contribution

It analyzes 20 recent papers to develop a set of observations for better conceptualizing and assessing the societal impact of cultural bias in NLP models.

Findings

Identified gaps in current bias evaluation methods.

Proposed a framework for societal impact assessment.

Highlighted the need for stakeholder engagement in bias evaluation.

Abstract

Research has shown that while large language models (LLMs) can generate their responses based on cultural context, they are not perfect and tend to generalize across cultures. However, when evaluating the cultural bias of a language technology on any dataset, researchers may choose not to engage with stakeholders actually using that technology in real life, which evades the very fundamental problem they set out to address. Inspired by the work done by arXiv:2005.14050v2, I set out to analyse recent literature about identifying and evaluating cultural bias in Natural Language Processing (NLP). I picked out 20 papers published in 2025 about cultural bias and came up with a set of observations to allow NLP researchers in the future to conceptualize bias concretely and evaluate its harms effectively. My aim is to advocate for a robust assessment of the societal impact of language technologies exhibiting cross-cultural bias.