Multi-Grained Patch Training for Efficient LLM-based Recommendation

TL;DR

This paper introduces PatchRec, a two-stage training method that enhances LLMs to model long-term user interaction histories efficiently for recommendation tasks by using aggregated embeddings and time-aware fine-tuning.

Contribution

It proposes a novel multi-grained patch training approach that enables LLMs to better understand long-term user behavior in recommendation systems.

Findings

Improved ability to model longer user histories.

Enhanced efficiency in training and inference.

Better recommendation performance on benchmark datasets.

Abstract

Large Language Models (LLMs) have emerged as a new paradigm for recommendation by converting interacted item history into language modeling. However, constrained by the limited context length of LLMs, existing approaches have to truncate item history in the prompt, focusing only on recent interactions and sacrificing the ability to model long-term history. To enable LLMs to model long histories, we pursue a concise embedding representation for items and sessions. In the LLM embedding space, we construct an item's embedding by aggregating its textual token embeddings; similarly, we construct a session's embedding by aggregating its item embeddings. While efficient, this way poses two challenges since it ignores the temporal significance of user interactions and LLMs do not natively interpret our custom embeddings. To overcome these, we propose PatchRec, a multi-grained patch training method consisting of two stages: (1) Patch Pre-training, which familiarizes LLMs with aggregated embeddings -- patches, and (2) Patch Fine-tuning, which enables LLMs to capture time-aware significance in interaction history. Extensive experiments show that PatchRec effectively models longer behavior histories with improved efficiency. This work facilitates the practical use of LLMs for modeling long behavior histories. Codes are available at https://github.com/ljy0ustc/PatchRec.