Do Multilingual Language Models Capture Differing Moral Norms?

TL;DR

This paper investigates whether multilingual language models encode varying moral norms across languages, highlighting potential biases from training data and exploring methods to detect and mitigate these issues.

Contribution

The study assesses how multilingual models capture moral norms across languages and proposes approaches to identify and address biases stemming from training data imbalances.

Findings

Multilingual models can encode moral norms, sometimes aligning with human agreement.

Differences in moral norm representations exist between languages.

Models trained on high-resource languages may impose their norms on low-resource languages.

Abstract

Massively multilingual sentence representations are trained on large corpora of uncurated data, with a very imbalanced proportion of languages included in the training. This may cause the models to grasp cultural values including moral judgments from the high-resource languages and impose them on the low-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs. Both these issues can negatively influence zero-shot cross-lingual model transfer and potentially lead to harmful outcomes. Therefore, we aim to (1) detect and quantify these issues by comparing different models in different languages, (2) develop methods for improving undesirable properties of the models. Our initial experiments using the multilingual model XLM-R show that indeed multilingual LMs capture moral norms, even with potentially higher human-agreement than monolingual ones. However, it is not yet clear to what extent these moral norms differ between languages.