Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions

TL;DR

This paper investigates religious bias in multilingual large language models, revealing persistent biases towards Islam and language-dependent performance disparities, especially between English and Bengali, in sensitive religious contexts.

Contribution

Introduces the BRAND dataset for evaluating religious bias in multilingual models and highlights language and religion-specific biases in model responses.

Findings

Models perform better in English than Bengali.

Models show bias towards Islam regardless of question neutrality.

Bias persists across different languages and religious contexts.

Abstract

While recent developments in large language models have improved bias detection and classification, sensitive subjects like religion still present challenges because even minor errors can result in severe misunderstandings. In particular, multilingual models often misrepresent religions and have difficulties being accurate in religious contexts. To address this, we introduce BRAND: Bilingual Religious Accountable Norm Dataset, which focuses on the four main religions of South Asia: Buddhism, Christianity, Hinduism, and Islam, containing over 2,400 entries, and we used three different types of prompts in both English and Bengali. Our results indicate that models perform better in English than in Bengali and consistently display bias toward Islam, even when answering religion-neutral questions. These findings highlight persistent bias in multilingual models when similar questions are asked in different languages. We further connect our findings to the broader issues in HCI regarding religion and spirituality.