Knowledge Distillation Approaches for Accurate and Efficient Recommender System

TL;DR

This paper introduces novel knowledge distillation methods tailored for recommender systems, enhancing the performance of compact models by transferring latent and ranking knowledge effectively, thus improving accuracy and efficiency.

Contribution

It develops new distillation techniques for recommender systems, including latent knowledge transfer, relation transfer, and ranking knowledge compression, addressing prior research gaps.

Findings

Improved recommendation accuracy with compact models.

Effective transfer of niche user preferences.

Reduced computational costs in ensemble models.

Abstract

Despite its breakthrough in classification problems, Knowledge distillation (KD) to recommendation models and ranking problems has not been studied well in the previous literature. This dissertation is devoted to developing knowledge distillation methods for recommender systems to fully improve the performance of a compact model. We propose novel distillation methods designed for recommender systems. The proposed methods are categorized according to their knowledge sources as follows: (1) Latent knowledge: we propose two methods that transfer latent knowledge of user/item representation. They effectively transfer knowledge of niche tastes with a balanced distillation strategy that prevents the KD process from being biased towards a small number of large preference groups. Also, we propose a new method that transfers user/item relations in the representation space. The proposed method selectively transfers essential relations considering the limited capacity of the compact model. (2) Ranking knowledge: we propose three methods that transfer ranking knowledge from the recommendation results. They formulate the KD process as a ranking matching problem and transfer the knowledge via a listwise learning strategy. Further, we present a new learning framework that compresses the ranking knowledge of heterogeneous recommendation models. The proposed framework is developed to ease the computational burdens of model ensemble which is a dominant solution for many recommendation applications. We validate the benefit of our proposed methods and frameworks through extensive experiments. To summarize, this dissertation sheds light on knowledge distillation approaches for a better accuracy-efficiency trade-off of the recommendation models.