Continual Collaborative Distillation for Recommender System

TL;DR

This paper introduces a continual collaborative distillation framework for recommender systems that enables both teacher and student models to adapt over data streams, improving efficiency and accuracy in dynamic environments.

Contribution

It presents a novel continual collaborative distillation method allowing dynamic adaptation of teacher and student models in non-stationary data streams for recommender systems.

Findings

CCD effectively adapts to new data in real-world datasets.

The framework maintains high recommendation accuracy.

Experimental results show improved efficiency and adaptability.

Abstract

Knowledge distillation (KD) has emerged as a promising technique for addressing the computational challenges associated with deploying large-scale recommender systems. KD transfers the knowledge of a massive teacher system to a compact student model, to reduce the huge computational burdens for inference while retaining high accuracy. The existing KD studies primarily focus on one-time distillation in static environments, leaving a substantial gap in their applicability to real-world scenarios dealing with continuously incoming users, items, and their interactions. In this work, we delve into a systematic approach to operating the teacher-student KD in a non-stationary data stream. Our goal is to enable efficient deployment through a compact student, which preserves the high performance of the massive teacher, while effectively adapting to continuously incoming data. We propose Continual Collaborative Distillation (CCD) framework, where both the teacher and the student continually and collaboratively evolve along the data stream. CCD facilitates the student in effectively adapting to new data, while also enabling the teacher to fully leverage accumulated knowledge. We validate the effectiveness of CCD through extensive quantitative, ablative, and exploratory experiments on two real-world datasets. We expect this research direction to contribute to narrowing the gap between existing KD studies and practical applications, thereby enhancing the applicability of KD in real-world systems.