The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora

TL;DR

This paper investigates retrieval biases in cross-lingual RAG systems over Arabic-English corpora, revealing retrieval as a key bottleneck and proposing strategies to improve multilingual retrieval performance in real-world scenarios.

Contribution

It provides a systematic analysis of multilingual retrieval challenges in domain-specific RAG, introduces new benchmarks, and proposes simple strategies to mitigate retrieval biases across languages.

Findings

Retrieval performance drops when user query and document languages differ.

Retrieval biases are mainly due to the retriever's difficulty in ranking cross-lingual documents.

Simple strategies like enforcing equal retrieval or translating queries improve performance.

Abstract

Cross-lingual retrieval-augmented generation (RAG) is a critical capability for retrieving and generating answers across languages. Prior work in this context has mostly focused on generation and relied on benchmarks derived from open-domain sources, most notably Wikipedia. In such settings, retrieval challenges often remain hidden due to language imbalances, overlap with pretraining data, and memorized content. To address this gap, we study Arabic-English RAG in a domain-specific setting using benchmarks derived from real-world corporate datasets. Our benchmarks include all combinations of languages for the user query and the supporting document, drawn independently and uniformly at random. This enables a systematic study of multilingual retrieval behavior. Our findings reveal that retrieval is a critical bottleneck in cross-lingual domain-specific scenarios, with substantial performance drops occurring when the user query and supporting document languages differ. A key insight is that these failures stem primarily from the retriever's difficulty in ranking documents across languages. Finally, we propose two simple retrieval strategies that address this source of failure by enforcing equal retrieval from both languages or by translating the query, resulting in substantial improvements in cross-lingual and overall performance. These results highlight meaningful opportunities for improving multilingual retrieval, particularly in practical, real-world RAG applications.