Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation

TL;DR

This paper introduces a pairwise ranking distillation method that efficiently mimics the performance of Pairwise Ranking Prompting (PRP) with significantly reduced computational costs, making high-quality document ranking more practical.

Contribution

The authors propose a novel distillation approach that transfers PRP's effectiveness to a pointwise student model, achieving similar performance with only 2% of the pairwise data.

Findings

The distilled student model matches PRP performance using only 2% of pairs.

The approach reduces computational costs during training and inference.

Sample-efficient distillation retains ranking quality with minimal data.

Abstract

While Pairwise Ranking Prompting (PRP) with Large Language Models (LLMs) is one of the most effective zero-shot document ranking methods, it has a quadratic computational complexity with respect to the number of documents to be ranked, as it requires an enumeration over all possible document pairs. Consequently, the outstanding ranking performance of PRP has remained unreachable for most real-world ranking applications. In this work, we propose to harness the effectiveness of PRP through pairwise distillation. Specifically, we distill a pointwise student ranker from pairwise teacher labels generated by PRP, resulting in an efficient student model that retains the performance of PRP with substantially lower computational costs. Furthermore, we find that the distillation process can be made sample-efficient: with only 2% of pairs, we are able to obtain the same performance as using all pairs for teacher labels. Thus, our novel approach provides a solution to harness the ranking performance of PRP without incurring high computational costs during both distillation and serving.