Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

TL;DR

This paper introduces Instance-wise LoRA (iLoRA), a novel method for sequential recommendation that dynamically customizes language model fine-tuning for individual user sequences, significantly improving accuracy with minimal additional parameters.

Contribution

The paper proposes iLoRA, integrating LoRA with MoE and a sequence-guided gating mechanism to better capture user behavior variability in sequential recommendation tasks.

Findings

iLoRA achieves 11.4% relative improvement in hit ratio over basic LoRA.

iLoRA requires less than 1% additional trainable parameters.

Extensive experiments validate the effectiveness of iLoRA across benchmark datasets.

Abstract

Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches are eager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4\% over basic LoRA in the hit ratio metric, with less than a 1\% relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code are available at https://github.com/AkaliKong/iLoRA.