Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System

TL;DR

This paper introduces ranking distillation, a novel knowledge distillation technique for learning compact, high-performance ranking models in recommender systems, achieving efficiency without sacrificing effectiveness.

Contribution

The paper proposes a new ranking distillation method that trains smaller models to mimic larger models, improving efficiency while maintaining ranking performance.

Findings

Student models are less than half the size of teacher models.

Ranking performance of student models is comparable to teacher models.

Ranking distillation outperforms models trained without it.

Abstract

We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectiveness and efficiency. We propose a KD technique for learning to rank problems, called \emph{ranking distillation (RD)}. Specifically, we train a smaller student model to learn to rank documents/items from both the training data and the supervision of a larger teacher model. The student model achieves a similar ranking performance to that of the large teacher model, but its smaller model size makes the online inference more efficient. RD is flexible because it is orthogonal to the choices of ranking models for the teacher and student. We address the challenges of RD for ranking problems. The experiments on public data sets and state-of-the-art recommendation models showed that RD achieves its design purposes: the student model learnt with RD has a model size less than half of the teacher model while achieving a ranking performance similar to the teacher model and much better than the student model learnt without RD.