Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation

TL;DR

This paper introduces a lightweight sequential recommendation algorithm that leverages pre-trained language models and knowledge distillation to improve accuracy and enable real-time recommendations.

Contribution

It proposes a two-stage method combining fine-tuning and knowledge distillation to create efficient, high-performing recommendation models based on pre-trained language models.

Findings

Enhanced recommendation accuracy demonstrated on multiple datasets.

Achieved lightweight models suitable for real-time recommendation systems.

Effective transfer of knowledge across domains via distillation.

Abstract

Sequential recommendation models user interests based on historical behaviors to provide personalized recommendation. Previous sequential recommendation algorithms primarily employ neural networks to extract features of user interests, achieving good performance. However, due to the recommendation system datasets sparsity, these algorithms often employ small-scale network frameworks, resulting in weaker generalization capability. Recently, a series of sequential recommendation algorithms based on large pre-trained language models have been proposed. Nonetheless, given the real-time demands of recommendation systems, the challenge remains in applying pre-trained language models for rapid recommendations in real scenarios. To address this, we propose a sequential recommendation algorithm based on a pre-trained language model and knowledge distillation. The key of proposed algorithm is to transfer pre-trained knowledge across domains and achieve lightweight inference by knowledge distillation. The algorithm operates in two stages: in the first stage, we fine-tune the pre-trained language model on the recommendation dataset to transfer the pre-trained knowledge to the recommendation task; in the second stage, we distill the trained language model to transfer the learned knowledge to a lightweight model. Extensive experiments on multiple public recommendation datasets show that the proposed algorithm enhances recommendation accuracy and provide timely recommendation services.