Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models

TL;DR

This paper introduces the MLNeedle test to evaluate how well multilingual large language models retrieve relevant information from long, complex, multilingual contexts, revealing significant performance variability across languages and input positions.

Contribution

It presents the first systematic evaluation of multilingual LLMs' long-context retrieval capabilities using the novel MLNeedle benchmark, highlighting key challenges and performance gaps.

Findings

Performance drops for non-English languages and middle-of-input needles.

Models struggle with cross-lingual retrieval as context length increases.

Significant variability in model performance based on language and position.

Abstract

While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model's ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of $8k$ tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.