HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling

TL;DR

This paper introduces HLLM, a hierarchical large language model architecture that significantly improves sequential recommendation performance by leveraging pre-trained LLMs for item content extraction and user interest prediction, demonstrating scalability and efficiency.

Contribution

The paper presents a novel two-tier hierarchical LLM architecture for recommendation systems, effectively utilizing pre-trained models and demonstrating scalability and practical efficiency.

Findings

HLLM outperforms traditional models on large-scale datasets.

Fine-tuning enhances HLLM's recommendation accuracy.

HLLM achieves state-of-the-art results with 7B parameter models.

Abstract

Large Language Models (LLMs) have achieved remarkable success in various fields, prompting several studies to explore their potential in recommendation systems. However, these attempts have so far resulted in only modest improvements over traditional recommendation models. Moreover, three critical questions remain under-explored: firstly, the real value of LLMs' pre-trained weights, often considered to encapsulate world knowledge; secondly, the necessity of fine-tuning for recommendation tasks; lastly, whether LLMs can exhibit the same scalability benefits in recommendation systems as they do in other domains. In this paper, we propose a novel Hierarchical Large Language Model (HLLM) architecture designed to enhance sequential recommendation systems. Our approach employs a two-tier model: the first Item LLM extracts rich content features from the detailed text description of the item, while the second User LLM utilizes these features to predict users' future interests based on their interaction history. Extensive experiments demonstrate that our method effectively leverages the pre-trained capabilities of open-source LLMs, and further fine-tuning leads to significant performance boosts. Additionally, HLLM achieves excellent scalability, with the largest configuration utilizing 7B parameters for both item feature extraction and user interest modeling. Moreover, HLLM offers excellent training and serving efficiency, making it practical in real-world applications. Evaluations on two large-scale datasets, PixelRec and Amazon Reviews, show that HLLM achieves state-of-the-art results, outperforming traditional ID-based models by a wide margin. In online A/B testing, HLLM showcases notable gains, validating its practical impact in real-world recommendation scenarios. Codes are available at https://github.com/bytedance/HLLM.