Do RAG Systems Really Suffer From Positional Bias?

TL;DR

This paper examines the impact of positional bias in Retrieval Augmented Generation systems, revealing that distracting passages often dominate retrieval results and that positional bias has limited effect in real-world scenarios.

Contribution

It provides empirical evidence that positional bias has minimal impact in practical RAG systems due to the prevalence of distracting passages in top retrievals.

Findings

Over 60% of queries have distracting passages among top-10 retrieved.

Rearranging passages based on positional bias does not improve performance.

Distracting passages significantly influence retrieval outcomes in RAG systems.

Abstract

Retrieval Augmented Generation enhances LLM accuracy by adding passages retrieved from an external corpus to the LLM prompt. This paper investigates how positional bias - the tendency of LLMs to weight information differently based on its position in the prompt - affects not only the LLM's capability to capitalize on relevant passages, but also its susceptibility to distracting passages. Through extensive experiments on three benchmarks, we show how state-of-the-art retrieval pipelines, while attempting to retrieve relevant passages, systematically bring highly distracting ones to the top ranks, with over 60% of queries containing at least one highly distracting passage among the top-10 retrieved passages. As a result, the impact of the LLM positional bias, which in controlled settings is often reported as very prominent by related works, is actually marginal in real scenarios since both relevant and distracting passages are, in turn, penalized. Indeed, our findings reveal that sophisticated strategies that attempt to rearrange the passages based on LLM positional preferences do not perform better than random shuffling.