Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

TL;DR

This paper introduces an adaptive early-stopping method to reduce the number of LLM calls in ranking tasks by mitigating position bias and repetition inconsistencies, achieving significant efficiency gains with minimal accuracy loss.

Contribution

The paper presents a novel dynamic repetition strategy with confidence-based adaptation that significantly decreases LLM calls while maintaining ranking accuracy.

Findings

Reduces LLM calls by up to 87% with confidence-based adaptation.

Maintains ranking accuracy despite fewer repetitions.

Effective across multiple models and tasks.

Abstract

When using LLMs to rank items based on given criteria, or evaluate answers, the order of candidate items can influence the model's final decision. This sensitivity to item positioning in a LLM's prompt is known as position bias. Prior research shows that this bias exists even in large models, though its severity varies across models and tasks. In addition to position bias, LLMs also exhibit varying degrees of low repetition consistency, where repeating the LLM call with the same candidate ordering can lead to different rankings. To address both inconsistencies, a common approach is to prompt the model multiple times with different candidate orderings and aggregate the results via majority voting. However, this repetition strategy, significantly increases computational costs. Extending prior findings, we observe that both the direction -- favoring either the earlier or later candidate in the prompt -- and magnitude of position bias across instances vary substantially, even within a single dataset. This observation highlights the need for a per-instance mitigation strategy. To this end, we introduce a dynamic early-stopping method that adaptively determines the number of repetitions required for each instance. Evaluating our approach across three LLMs of varying sizes and on two tasks, namely re-ranking and alignment, we demonstrate that transitioning to a dynamic repetition strategy reduces the number of LLM calls by an average of 81%, while preserving the accuracy. Furthermore, we propose a confidence-based adaptation to our early-stopping method, reducing LLM calls by an average of 87% compared to static repetition, with only a slight accuracy trade-off relative to our original early-stopping method.