Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation

TL;DR

This paper introduces ReLLaX, a comprehensive framework that enhances large language models for lifelong sequential behavior understanding in recommendation systems by optimizing data, prompts, and model parameters.

Contribution

The paper proposes ReLLaX, integrating SUBR, SPA, and CFLoRA to improve LLMs' ability to process long user behavior sequences for recommendation tasks, with theoretical and empirical validation.

Findings

ReLLaX outperforms existing baselines on three public datasets.

SUBR reduces sequence heterogeneity, aiding information extraction.

CFLoRA enhances model expressiveness and captures sequential info better.

Abstract

In this paper, we address the lifelong sequential behavior incomprehension problem in large language models (LLMs) for recommendation, where LLMs struggle to extract useful information from long user behavior sequences, even within their context limits. To tackle this, we propose ReLLaX (Retrieval-enhanced Large Language models Plus), a framework offering optimization across data, prompt, and parameter levels. At the data level, we introduce Semantic User Behavior Retrieval (SUBR) to reduce sequence heterogeneity, making it easier for LLMs to extract key information. For prompt-level enhancement, we employ Soft Prompt Augmentation (SPA) to inject collaborative knowledge, aligning item representations with recommendation tasks and improving LLMs's exploration of item relationships. Finally, at the parameter level, we propose Component Fully-interactive LoRA (CFLoRA), which enhances LoRA's expressiveness by enabling interactions between its components, allowing better capture of sequential information. Moreover, we present new perspectives to compare current LoRA-based LLM4Rec methods, i.e. from both a composite and a decomposed view. We theoretically demonstrate that the ways they employ LoRA for recommendation are degraded versions of our CFLoRA, with different constraints on atom component interactions. Extensive experiments on three public datasets demonstrate ReLLaX's superiority over existing baselines and its ability to mitigate lifelong sequential behavior incomprehension effectively.