Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models

TL;DR

This study reveals that multilingual large language models tend to favor information in the same language as the query and prefer high-resource languages when no direct information is available, potentially reinforcing language bias.

Contribution

The paper uncovers systemic linguistic biases in multilingual LLMs during information retrieval and generation, highlighting a disparity that affects information parity across languages.

Findings

LLMs prefer same-language documents in retrieval and answer generation.

Bias towards high-resource languages when no information is in the query language.

Linguistic bias exists for both factual and opinion-based queries.

Abstract

Although the multilingual capability of LLMs offers new opportunities to overcome the language barrier, do these capabilities translate into real-life scenarios where linguistic divide and knowledge conflicts between multilingual sources are known occurrences? In this paper, we studied LLM's linguistic preference in a cross-language RAG-based information search setting. We found that LLMs displayed systemic bias towards information in the same language as the query language in both document retrieval and answer generation. Furthermore, in scenarios where no information is in the language of the query, LLMs prefer documents in high-resource languages during generation, potentially reinforcing the dominant views. Such bias exists for both factual and opinion-based queries. Our results highlight the linguistic divide within multilingual LLMs in information search systems. The seemingly beneficial multilingual capability of LLMs may backfire on information parity by reinforcing language-specific information cocoons or filter bubbles further marginalizing low-resource views.