Sparse Pairwise Re-ranking with Pre-trained Transformers

TL;DR

This paper explores efficient pairwise re-ranking using sampling methods with pre-trained transformers, reducing comparisons significantly while maintaining competitive relevance ranking performance.

Contribution

It introduces sampling strategies for pairwise re-ranking that drastically reduce comparisons needed, improving efficiency without sacrificing much effectiveness.

Findings

Order of magnitude fewer comparisons needed

Competitive effectiveness with only one third of comparisons

Sampling methods outperform full pairwise comparisons

Abstract

Pairwise re-ranking models predict which of two documents is more relevant to a query and then aggregate a final ranking from such preferences. This is often more effective than pointwise re-ranking models that directly predict a relevance value for each document. However, the high inference overhead of pairwise models limits their practical application: usually, for a set of $k$ documents to be re-ranked, preferences for all $k^2-k$ comparison pairs excluding self-comparisons are aggregated. We investigate whether the efficiency of pairwise re-ranking can be improved by sampling from all pairs. In an exploratory study, we evaluate three sampling methods and five preference aggregation methods. The best combination allows for an order of magnitude fewer comparisons at an acceptable loss of retrieval effectiveness, while competitive effectiveness is already achieved with about one third of the comparisons.