Cross-Language Bias Examination in Large Language Models

TL;DR

This paper presents a new framework for evaluating bias in multilingual large language models, revealing significant cross-lingual bias variations and emphasizing the importance of implicit bias detection.

Contribution

It introduces a comprehensive multilingual bias evaluation framework combining explicit and implicit bias assessments across five languages, filling a key research gap.

Findings

Arabic and Spanish show higher stereotype bias

Chinese and English exhibit lower bias levels

Implicit bias is often higher than explicit bias in age-related assessments

Abstract

This study introduces an innovative multilingual bias evaluation framework for assessing bias in Large Language Models, combining explicit bias assessment through the BBQ benchmark with implicit bias measurement using a prompt-based Implicit Association Test. By translating the prompts and word list into five target languages, English, Chinese, Arabic, French, and Spanish, we directly compare different types of bias across languages. The results reveal substantial gaps in bias across languages used in LLMs. For example, Arabic and Spanish consistently show higher levels of stereotype bias, while Chinese and English exhibit lower levels of bias. We also identify contrasting patterns across bias types. Age shows the lowest explicit bias but the highest implicit bias, emphasizing the importance of detecting implicit biases that are undetectable with standard benchmarks. These findings indicate that LLMs vary significantly across languages and bias dimensions. This study fills a key research gap by providing a comprehensive methodology for cross-lingual bias analysis. Ultimately, our work establishes a foundation for the development of equitable multilingual LLMs, ensuring fairness and effectiveness across diverse languages and cultures.