An Empirical Study of Position Bias in Modern Information Retrieval

TL;DR

This paper empirically examines position bias in modern information retrieval models, introducing new benchmarks and metrics to evaluate how models prioritize early passage content over later relevant information, revealing varying robustness among different retrieval methods.

Contribution

The study introduces two position-aware retrieval benchmarks and the Position Sensitivity Index (PSI) to quantify position bias, providing a comprehensive evaluation of multiple retrieval models' sensitivity to passage position.

Findings

Dense embedding and ColBERT models degrade by 15.6% when relevant info is later in passages.

BM25 and reranker models are more robust to positional variations.

The evaluation framework helps identify and improve position bias in IR systems.

Abstract

This study investigates the position bias in information retrieval, where models tend to overemphasize content at the beginning of passages while neglecting semantically relevant information that appears later. To analyze the extent and impact of position bias, we introduce a new evaluation framework consisting of two position-aware retrieval benchmarks (SQuAD-PosQ, FineWeb-PosQ) and an intuitive diagnostic metric, the Position Sensitivity Index (PSI), for quantifying position bias from a worst-case perspective. We conduct a comprehensive evaluation across the full retrieval pipeline, including BM25, dense embedding models, ColBERT-style late-interaction models, and full-interaction reranker models. Our experiments show that when relevant information appears later in the passage, dense embedding models and ColBERT-style models suffer significant performance degradation (an average drop of 15.6%). In contrast, BM25 and reranker models demonstrate greater robustness to such positional variation. These findings provide practical insights into model sensitivity to the position of relevant information and offer guidance for building more position-robust retrieval systems. Code and data are publicly available at: https://github.com/NovaSearch-Team/position-bias-in-IR.