Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking

TL;DR

This paper introduces Rank-DistiLLM, a new dataset and training approach that significantly improves the effectiveness of distilled cross-encoders for passage re-ranking, making them comparable to large language models while being much faster and more efficient.

Contribution

The paper presents Rank-DistiLLM, a novel dataset and training methodology that closes the effectiveness gap between distilled cross-encoders and LLMs in passage re-ranking.

Findings

Cross-encoders trained on Rank-DistiLLM match LLM effectiveness.

Distilled models are up to 173 times faster and 24 times more memory efficient.

Applying advanced fine-tuning methods improves cross-encoder performance.

Abstract

Cross-encoders distilled from large language models (LLMs) are often more effective re-rankers than cross-encoders fine-tuned on manually labeled data. However, distilled models do not match the effectiveness of their teacher LLMs. We hypothesize that this effectiveness gap is due to the fact that previous work has not applied the best-suited methods for fine-tuning cross-encoders on manually labeled data (e.g., hard-negative sampling, deep sampling, and listwise loss functions). To close this gap, we create a new dataset, Rank-DistiLLM. Cross-encoders trained on Rank-DistiLLM achieve the effectiveness of LLMs while being up to 173 times faster and 24 times more memory efficient. Our code and data is available at https://github.com/webis-de/ECIR-25.