Rethinking Large Language Model Architectures for Sequential Recommendations

TL;DR

This paper introduces Lite-LLM4Rec, a streamlined large language model architecture for sequential recommendation that significantly reduces inference costs while maintaining high performance, by eliminating beam search and employing a hierarchical structure.

Contribution

The paper proposes Lite-LLM4Rec, a novel LLM-based recommendation model that improves inference efficiency and effectiveness through a simple item projection head and hierarchical design.

Findings

Achieves 46.8% performance improvement on ML-1m dataset.

Reduces inference time by 97.28% compared to existing methods.

Maintains high recommendation quality with simplified architecture.

Abstract

Recently, sequential recommendation has been adapted to the LLM paradigm to enjoy the power of LLMs. LLM-based methods usually formulate recommendation information into natural language and the model is trained to predict the next item in an auto-regressive manner. Despite their notable success, the substantial computational overhead of inference poses a significant obstacle to their real-world applicability. In this work, we endeavor to streamline existing LLM-based recommendation models and propose a simple yet highly effective model Lite-LLM4Rec. The primary goal of Lite-LLM4Rec is to achieve efficient inference for the sequential recommendation task. Lite-LLM4Rec circumvents the beam search decoding by using a straight item projection head for ranking scores generation. This design stems from our empirical observation that beam search decoding is ultimately unnecessary for sequential recommendations. Additionally, Lite-LLM4Rec introduces a hierarchical LLM structure tailored to efficiently handle the extensive contextual information associated with items, thereby reducing computational overhead while enjoying the capabilities of LLMs. Experiments on three publicly available datasets corroborate the effectiveness of Lite-LLM4Rec in both performance and inference efficiency (notably 46.8% performance improvement and 97.28% efficiency improvement on ML-1m) over existing LLM-based methods. Our implementations will be open sourced.