LLMs as mirrors of societal moral standards: reflection of cultural divergence and agreement across ethical topics

TL;DR

This study evaluates whether large language models accurately mirror cross-cultural moral standards, revealing limited performance in capturing cultural divergences and agreements, which underscores the need for improved bias mitigation and cultural sensitivity.

Contribution

It introduces a systematic evaluation framework for assessing LLMs' ability to reflect cultural moral variations using multiple analytical methods.

Findings

LLMs show variable and generally low accuracy in reflecting cross-cultural moral differences.

Current LLMs do not reliably capture cultural similarities in moral perspectives.

The study highlights the need for enhancing models to better represent diverse cultural values.

Abstract

Large language models (LLMs) have become increasingly pivotal in various domains due the recent advancements in their performance capabilities. However, concerns persist regarding biases in LLMs, including gender, racial, and cultural biases derived from their training data. These biases raise critical questions about the ethical deployment and societal impact of LLMs. Acknowledging these concerns, this study investigates whether LLMs accurately reflect cross-cultural variations and similarities in moral perspectives. In assessing whether the chosen LLMs capture patterns of divergence and agreement on moral topics across cultures, three main methods are employed: (1) comparison of model-generated and survey-based moral score variances, (2) cluster alignment analysis to evaluate the correspondence between country clusters derived from model-generated moral scores and those derived from survey data, and (3) probing LLMs with direct comparative prompts. All three methods involve the use of systematic prompts and token pairs designed to assess how well LLMs understand and reflect cultural variations in moral attitudes. The findings of this study indicate overall variable and low performance in reflecting cross-cultural differences and similarities in moral values across the models tested, highlighting the necessity for improving models' accuracy in capturing these nuances effectively. The insights gained from this study aim to inform discussions on the ethical development and deployment of LLMs in global contexts, emphasizing the importance of mitigating biases and promoting fair representation across diverse cultural perspectives.