Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation

TL;DR

This paper introduces Pos2Distill, a knowledge distillation framework that mitigates position bias in long-context tasks by transferring capabilities from advantageous to less favorable positions, improving uniformity and performance.

Contribution

The paper proposes a novel position to position knowledge distillation method, Pos2Distill, to effectively reduce position bias without high data or computational costs.

Findings

Significant performance improvements across all positions in retrieval and reasoning tasks.

Enhanced uniformity in long-context comprehension.

Strong cross-task generalization of the proposed method.

Abstract

Positional bias (PB), manifesting as non-uniform sensitivity across different contextual locations, significantly impairs long-context comprehension and processing capabilities. Previous studies have addressed PB either by modifying the underlying architectures or by employing extensive contextual awareness training. However, the former approach fails to effectively eliminate the substantial performance disparities, while the latter imposes significant data and computational overhead. To address PB effectively, we introduce \textbf{Pos2Distill}, a position to position knowledge distillation framework. Pos2Distill transfers the superior capabilities from advantageous positions to less favorable ones, thereby reducing the huge performance gaps. The conceptual principle is to leverage the inherent, position-induced disparity to counteract the PB itself. We identify distinct manifestations of PB under \textbf{\textsc{r}}etrieval and \textbf{\textsc{r}}easoning paradigms, thereby designing two specialized instantiations: \emph{Pos2Distill-R\textsuperscript{1}} and \emph{Pos2Distill-R\textsuperscript{2}} respectively, both grounded in this core principle. By employing the Pos2Distill approach, we achieve enhanced uniformity and significant performance gains across all contextual positions in long-context retrieval and reasoning tasks. Crucially, both specialized systems exhibit strong cross-task generalization mutually, while achieving superior performance on their respective tasks.