Collaborative Distillation for Top-N Recommendation

TL;DR

This paper introduces a collaborative distillation method for top-N recommendation that effectively compresses recommender models, addressing feedback ambiguity and ranking challenges, and outperforms existing methods in key metrics.

Contribution

It proposes a novel collaborative distillation framework with specialized loss functions and training strategies tailored for top-N recommendation tasks.

Findings

Outperforms state-of-the-art methods by up to 33.2% in hit rate.

Achieves comparable performance to teacher models.

Effectively handles feedback ambiguity and ranking issues.

Abstract

Knowledge distillation (KD) is a well-known method to reduce inference latency by compressing a cumbersome teacher model to a small student model. Despite the success of KD in the classification task, applying KD to recommender models is challenging due to the sparsity of positive feedback, the ambiguity of missing feedback, and the ranking problem associated with the top-N recommendation. To address the issues, we propose a new KD model for the collaborative filtering approach, namely collaborative distillation (CD). Specifically, (1) we reformulate a loss function to deal with the ambiguity of missing feedback. (2) We exploit probabilistic rank-aware sampling for the top-N recommendation. (3) To train the proposed model effectively, we develop two training strategies for the student model, called the teacher- and the student-guided training methods, selecting the most useful feedback from the teacher model. Via experimental results, we demonstrate that the proposed model outperforms the state-of-the-art method by 2.7-33.2% and 2.7-29.1% in hit rate (HR) and normalized discounted cumulative gain (NDCG), respectively. Moreover, the proposed model achieves the performance comparable to the teacher model.