Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models

TL;DR

This paper investigates ethical biases in large language models across multiple languages using the newly created MSQAD dataset and statistical hypothesis tests, revealing widespread cross-language biases.

Contribution

Introduces the MSQAD dataset for multilingual bias analysis and applies statistical tests to demonstrate the prevalence of ethical biases across languages and models.

Findings

Biases are prevalent across different languages and topics.

Cross-language differences significantly contribute to ethical biases.

Biases are consistent across various large language models.

Abstract

Despite the recent strides in large language models, studies have underscored the existence of social biases within these systems. In this paper, we delve into the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics, hypothesizing that these biases may arise from language-specific distinctions. Introducing the Multilingual Sensitive Questions & Answers Dataset (MSQAD), we collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages. We scrutinized the biases of these responses across languages and topics, employing two statistical hypothesis tests. The results showed that the null hypotheses were rejected in most cases, indicating biases arising from cross-language differences. It demonstrates that ethical biases in responses are widespread across various languages, and notably, these biases were prevalent even among different LLMs. By making the proposed MSQAD openly available, we aim to facilitate future research endeavors focused on examining cross-language biases in LLMs and their variant models.