Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

TL;DR

This paper introduces a knowledge distillation method that leverages textual user profiles from pre-trained LLMs to improve sequential recommenders, achieving rich user understanding without increasing inference costs.

Contribution

It proposes a novel approach to incorporate LLM-derived user profiles into sequential recommenders without needing LLM inference during deployment.

Findings

Maintains inference efficiency of traditional models.

No architectural modifications or LLM fine-tuning required.

Enhances user understanding in recommender systems.

Abstract

Sequential recommender systems have achieved significant success in modeling temporal user behavior but remain limited in capturing rich user semantics beyond interaction patterns. Large Language Models (LLMs) present opportunities to enhance user understanding with their reasoning capabilities, yet existing integration approaches create prohibitive inference costs in real time. To address these limitations, we present a novel knowledge distillation method that utilizes textual user profile generated by pre-trained LLMs into sequential recommenders without requiring LLM inference at serving time. The resulting approach maintains the inference efficiency of traditional sequential models while requiring neither architectural modifications nor LLM fine-tuning.