Cross-Task Knowledge Distillation in Multi-Task Recommendation

TL;DR

This paper introduces a novel Cross-Task Knowledge Distillation framework for multi-task recommendation systems, leveraging task-specific prediction results to improve overall performance through auxiliary tasks, calibrated distillation, and error correction mechanisms.

Contribution

It proposes a new framework combining MTL and KD, with auxiliary tasks, calibrated knowledge transfer, and error correction to address task conflicts and improve recommendation accuracy.

Findings

Enhanced recommendation performance on real-world datasets.

Effective knowledge transfer between tasks improves fine-grained user preference modeling.

The framework outperforms existing multi-task learning methods.

Abstract

Multi-task learning (MTL) has been widely used in recommender systems, wherein predicting each type of user feedback on items (e.g, click, purchase) are treated as individual tasks and jointly trained with a unified model. Our key observation is that the prediction results of each task may contain task-specific knowledge about user's fine-grained preference towards items. While such knowledge could be transferred to benefit other tasks, it is being overlooked under the current MTL paradigm. This paper, instead, proposes a Cross-Task Knowledge Distillation framework that attempts to leverage prediction results of one task as supervised signals to teach another task. However, integrating MTL and KD in a proper manner is non-trivial due to several challenges including task conflicts, inconsistent magnitude and requirement of synchronous optimization. As countermeasures, we 1) introduce auxiliary tasks with quadruplet loss functions to capture cross-task fine-grained ranking information and avoid task conflicts, 2) design a calibrated distillation approach to align and distill knowledge from auxiliary tasks, and 3) propose a novel error correction mechanism to enable and facilitate synchronous training of teacher and student models. Comprehensive experiments are conducted to verify the effectiveness of our framework in real-world datasets.