Lost in Sequence: Do Large Language Models Understand Sequential Recommendation?

TL;DR

This paper investigates whether large language models truly understand sequential recommendation tasks and introduces a new method, LLM-SRec, that improves sequence understanding and recommendation accuracy by distilling user representations.

Contribution

The paper reveals limitations of existing LLM-based recommendation models in capturing sequential information and proposes LLM-SRec, a simple method that enhances sequence understanding without extensive fine-tuning.

Findings

Existing LLM4Rec models do not fully capture sequential information.

LLM-SRec improves sequence understanding and recommendation performance.

State-of-the-art results achieved with minimal training of lightweight modules.

Abstract

Large Language Models (LLMs) have recently emerged as promising tools for recommendation thanks to their advanced textual understanding ability and context-awareness. Despite the current practice of training and evaluating LLM-based recommendation (LLM4Rec) models under a sequential recommendation scenario, we found that whether these models understand the sequential information inherent in users' item interaction sequences has been largely overlooked. In this paper, we first demonstrate through a series of experiments that existing LLM4Rec models do not fully capture sequential information both during training and inference. Then, we propose a simple yet effective LLM-based sequential recommender, called LLM-SRec, a method that enhances the integration of sequential information into LLMs by distilling the user representations extracted from a pre-trained CF-SRec model into LLMs. Our extensive experiments show that LLM-SRec enhances LLMs' ability to understand users' item interaction sequences, ultimately leading to improved recommendation performance. Furthermore, unlike existing LLM4Rec models that require fine-tuning of LLMs, LLM-SRec achieves state-of-the-art performance by training only a few lightweight MLPs, highlighting its practicality in real-world applications. Our code is available at https://github.com/Sein-Kim/LLM-SRec.