Probing Pre-Trained Language Models for Cross-Cultural Differences in Values

TL;DR

This paper investigates how pre-trained language models encode cultural values, revealing they capture cross-cultural differences but only weakly align with established surveys, highlighting challenges and opportunities for alignment.

Contribution

Introduces probes to analyze cultural values in language models and compares their embeddings with existing cross-cultural value surveys.

Findings

PTLMs encode cross-cultural value differences

Weak alignment between model embeddings and surveys

Implications for using PTLMs in multicultural contexts

Abstract

Language embeds information about social, cultural, and political values people hold. Prior work has explored social and potentially harmful biases encoded in Pre-Trained Language models (PTLMs). However, there has been no systematic study investigating how values embedded in these models vary across cultures. In this paper, we introduce probes to study which values across cultures are embedded in these models, and whether they align with existing theories and cross-cultural value surveys. We find that PTLMs capture differences in values across cultures, but those only weakly align with established value surveys. We discuss implications of using mis-aligned models in cross-cultural settings, as well as ways of aligning PTLMs with value surveys.