LLM-based Listwise Reranking under the Effect of Positional Bias

TL;DR

This paper introduces DebiasFirst, a method to mitigate positional bias in LLM-based listwise reranking by combining positional calibration and position-aware data augmentation, improving ranking robustness.

Contribution

DebiasFirst is a novel approach that effectively reduces positional bias in LLM reranking through integrated calibration and augmentation techniques.

Findings

Reduces dependence of NDCG@10 on relevant document position

Enhances effectiveness and robustness across diverse retrievers

Complements inference-stage debiasing methods

Abstract

LLM-based listwise passage reranking has attracted attention for its effectiveness in ranking candidate passages. However, these models suffer from positional bias, where passages positioned towards the end of the input are less likely to be moved to top positions in the ranking. We hypothesize that there are two primary sources of positional bias: (1) architectural bias inherent in LLMs and (2) the imbalanced positioning of relevant documents. To address this, we propose DebiasFirst, a method that integrates positional calibration and position-aware data augmentation during fine-tuning. Positional calibration uses inverse propensity scoring to adjust for positional bias by re-weighting the contributions of different positions in the loss function when training. Position-aware augmentation augments training data to ensure that each passage appears equally across varied positions in the input list. This approach markedly enhances both effectiveness and robustness to the original ranking across diverse first-stage retrievers, reducing the dependence of NDCG@10 performance on the position of relevant documents. DebiasFirst also complements the inference-stage debiasing methods, offering a practical solution for mitigating positional bias in reranking.