Beyong Tokens: Item-aware Attention for LLM-based Recommendation

TL;DR

This paper introduces an item-aware attention framework for LLM-based recommendation, emphasizing item-level relations over token-level, to improve recommendation accuracy.

Contribution

It proposes a novel item-aware attention mechanism with intra- and inter-item layers, explicitly modeling item content and collaborative relations in LLMs.

Findings

IAM improves recommendation performance on public datasets.

Explicit item-level modeling enhances LLM's ability to capture collaborative relations.

The framework outperforms existing token-centric attention methods.

Abstract

Large Language Models (LLMs) have recently gained increasing attention in the field of recommendation. Existing LLM-based methods typically represent items as token sequences, and apply attention layers on these tokens to generate recommendations. However, by inheriting the standard attention mechanism, these methods focus on modeling token-level relations. This token-centric focus overlooks the item as the fundamental unit of recommendation, preventing existing methods from effectively capturing collaborative relations at the item level. In this work, we revisit the role of tokens in LLM-driven recommendation and categorize their relations into two types: (1) intra-item token relations, which present the content semantics of an item, e.g., name, color, and size; and (2) inter-item token relations, which encode collaborative relations across items. Building on these insights, we propose a novel framework with an item-aware attention mechanism (IAM) to enhance LLMs for recommendation. Specifically, IAM devises two complementary attention layers: (1) an intra-item attention layer, which restricts attention to tokens within the same item, modeling item content semantics; and (2) an inter-item attention layer, which attends exclusively to token relations across items, capturing item collaborative relations. Through this stacked design, IAM explicitly emphasizes items as the fundamental units in recommendation, enabling LLMs to effectively exploit item-level collaborative relations. Extensive experiments on several public datasets demonstrate the effectiveness of IAM in enhancing LLMs for personalized recommendation.