ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting

TL;DR

ReAttn enhances attention-based re-ranking by re-weighting attention to reduce lexical bias and over-concentration, leading to more accurate and fair document relevance assessments without extra training.

Contribution

It introduces a post-hoc attention re-weighting method that improves relevance ranking by addressing attention concentration and lexical bias without additional training.

Findings

Significant improvement in re-ranking accuracy across multiple datasets

Reduction in lexical bias and attention over-concentration

No additional training required for the method

Abstract

The strong capabilities of recent Large Language Models (LLMs) have made them highly effective for zero-shot re-ranking task. Attention-based re-ranking methods, which derive relevance scores directly from attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First, attention signals are highly concentrated a small subset of tokens within a few documents, making others indistinguishable. Second, attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose \textbf{ReAttn}, a post-hoc re-weighting strategy for attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight attention on query-overlapping tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs entropy-based regularization to mitigate over-concentrated attention, encouraging a more balanced distribution across informative tokens. Both adjustments operate directly on existing attention weights without additional training or supervision. Extensive experiments demonstrate the effectiveness of our method.