Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation

TL;DR

This paper evaluates multilingual Retrieval-Augmented Language Models using a new benchmark, revealing linguistic inequalities and offering strategies to improve multilingual knowledge retrieval and generation.

Contribution

Introduces Futurepedia, a multilingual benchmark, and provides comprehensive analysis of challenges and biases in multilingual RALMs, guiding future improvements.

Findings

High-resource languages excel in knowledge extraction.

Indo-European languages facilitate answer generation from documents.

English dominates in knowledge selection due to bias.

Abstract

RALMs (Retrieval-Augmented Language Models) broaden their knowledge scope by incorporating external textual resources. However, the multilingual nature of global knowledge necessitates RALMs to handle diverse languages, a topic that has received limited research focus. In this work, we propose \textit{Futurepedia}, a carefully crafted benchmark containing parallel texts across eight representative languages. We evaluate six multilingual RALMs using our benchmark to explore the challenges of multilingual RALMs. Experimental results reveal linguistic inequalities: 1) high-resource languages stand out in Monolingual Knowledge Extraction; 2) Indo-European languages lead RALMs to provide answers directly from documents, alleviating the challenge of expressing answers across languages; 3) English benefits from RALMs' selection bias and speaks louder in multilingual knowledge selection. Based on these findings, we offer advice for improving multilingual Retrieval Augmented Generation. For monolingual knowledge extraction, careful attention must be paid to cascading errors from translating low-resource languages into high-resource ones. In cross-lingual knowledge transfer, encouraging RALMs to provide answers within documents in different languages can improve transfer performance. For multilingual knowledge selection, incorporating more non-English documents and repositioning English documents can help mitigate RALMs' selection bias. Through comprehensive experiments, we underscore the complexities inherent in multilingual RALMs and offer valuable insights for future research.