Improving Neural Ranking via Lossless Knowledge Distillation

TL;DR

This paper introduces Self-Distilled neural Rankers (SDR), a novel ranking method that significantly improves neural ranking performance by using a specialized listwise distillation framework and score transformation, surpassing traditional models.

Contribution

The paper proposes SDR, a new neural ranking approach that enhances performance without increasing model size, utilizing a unique listwise distillation and score transformation tailored for ranking.

Findings

SDR outperforms teacher models on 7 of 9 key metrics.

SDR surpasses gradient boosted decision trees in ranking tasks.

Theoretical analysis explains the effectiveness of listwise distillation.

Abstract

We explore a novel perspective of knowledge distillation (KD) for learning to rank (LTR), and introduce Self-Distilled neural Rankers (SDR), where student rankers are parameterized identically to their teachers. Unlike the existing ranking distillation work which pursues a good trade-off between performance and efficiency, SDR is able to significantly improve ranking performance of students over the teacher rankers without increasing model capacity. The key success factors of SDR, which differs from common distillation techniques for classification are: (1) an appropriate teacher score transformation function, and (2) a novel listwise distillation framework. Both techniques are specifically designed for ranking problems and are rarely studied in the existing knowledge distillation literature. Building upon the state-of-the-art neural ranking structure, SDR is able to push the limits of neural ranking performance above a recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics, the first time in the literature. In addition to the strong empirical results, we give theoretical explanations on why listwise distillation is effective for neural rankers, and provide ablation studies to verify the necessity of the key factors in the SDR framework.