Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

TL;DR

This paper investigates how the position of multiple relevant information pieces affects large language models' ability to process long inputs, revealing significant biases related to spacing that impact their effectiveness.

Contribution

The paper introduces LongPiBench, a benchmark for evaluating positional bias with multiple relevant pieces, and provides experimental insights into biases in current models.

Findings

Most models are robust against the 'lost in the middle' issue.

Significant biases exist related to spacing of relevant information.

Evaluating and reducing positional biases is crucial for LLM improvement.

Abstract

Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.