Laser: Parameter-Efficient LLM Bi-Tuning for Sequential Recommendation with Collaborative Information

TL;DR

Laser introduces a parameter-efficient bi-tuning framework for LLM-based sequential recommendation, effectively integrating collaborative information and user diversity while significantly outperforming existing methods.

Contribution

The paper proposes a novel Bi-Tuning approach with virtual tokens and a lightweight M-Former to enhance LLM-based recommendation accuracy and efficiency.

Findings

Laser outperforms state-of-the-art recommendation methods on real-world datasets.

The framework effectively incorporates collaborative information via prefix tuning.

M-Former improves user-specific recommendation accuracy.

Abstract

Sequential recommender systems are essential for discerning user preferences from historical interactions and facilitating targeted recommendations. Recent innovations employing Large Language Models (LLMs) have advanced the field by encoding item semantics, yet they often necessitate substantial parameter tuning and are resource-demanding. Moreover, these works fails to consider the diverse characteristics of different types of users and thus diminishes the recommendation accuracy. In this paper, we propose a parameter-efficient Large Language Model Bi-Tuning framework for sequential recommendation with collaborative information (Laser). Specifically, Bi-Tuning works by inserting trainable virtual tokens at both the prefix and suffix of the input sequence and freezing the LLM parameters, thus optimizing the LLM for the sequential recommendation. In our Laser, the prefix is utilized to incorporate user-item collaborative information and adapt the LLM to the recommendation task, while the suffix converts the output embeddings of the LLM from the language space to the recommendation space for the follow-up item recommendation. Furthermore, to capture the characteristics of different types of users when integrating the collaborative information via the prefix, we introduce M-Former, a lightweight MoE-based querying transformer that uses a set of query experts to integrate diverse user-specific collaborative information encoded by frozen ID-based sequential recommender systems, significantly improving the accuracy of recommendations. Extensive experiments on real-world datasets demonstrate that Laser can parameter-efficiently adapt LLMs to effective recommender systems, significantly outperforming state-of-the-art methods.