Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation

TL;DR

Tiny-NewsRec enhances news recommendation by adapting pre-trained language models to the news domain and distilling knowledge to create a more efficient and effective model suitable for real-time applications.

Contribution

The paper introduces a domain-specific post-training and a two-stage knowledge distillation approach for PLM-based news recommendation.

Findings

Improves recommendation accuracy in real-world datasets.

Reduces model size and computational overhead.

Maintains high performance after distillation.

Abstract

News recommendation is a widely adopted technique to provide personalized news feeds for the user. Recently, pre-trained language models (PLMs) have demonstrated the great capability of natural language understanding and benefited news recommendation via improving news modeling. However, most existing works simply finetune the PLM with the news recommendation task, which may suffer from the known domain shift problem between the pre-training corpus and downstream news texts. Moreover, PLMs usually contain a large volume of parameters and have high computational overhead, which imposes a great burden on low-latency online services. In this paper, we propose Tiny-NewsRec, which can improve both the effectiveness and the efficiency of PLM-based news recommendation. We first design a self-supervised domain-specific post-training method to better adapt the general PLM to the news domain with a contrastive matching task between news titles and news bodies. We further propose a two-stage knowledge distillation method to improve the efficiency of the large PLM-based news recommendation model while maintaining its performance. Multiple teacher models originated from different time steps of our post-training procedure are used to transfer comprehensive knowledge to the student in both its post-training and finetuning stage. Extensive experiments on two real-world datasets validate the effectiveness and efficiency of our method.