GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenization

TL;DR

GRACE is a novel generative recommendation framework that uses journey-aware sparse attention and explicit product attribute encoding to improve multi-behavior sequence modeling, interpretability, and computational efficiency.

Contribution

It introduces a hybrid Chain-of-Thought tokenization with explicit attributes and a journey-aware sparse attention mechanism for efficient, interpretable recommendations.

Findings

Outperforms state-of-the-art baselines by up to 106.9% in HR@10.

Reduces attention computation by up to 48%.

Achieves significant improvements in recommendation accuracy on real-world datasets.

Abstract

Generative models have recently demonstrated strong potential in multi-behavior recommendation systems, leveraging the expressive power of transformers and tokenization to generate personalized item sequences. However, their adoption is hindered by (1) the lack of explicit information for token reasoning, (2) high computational costs due to quadratic attention complexity and dense sequence representations after tokenization, and (3) limited multi-scale modeling over user history. In this work, we propose GRACE (Generative Recommendation via journey-aware sparse Attention on Chain-of-thought tokEnization), a novel generative framework for multi-behavior sequential recommendation. GRACE introduces a hybrid Chain-of-Thought (CoT) tokenization method that encodes user-item interactions with explicit attributes from product knowledge graphs (e.g., category, brand, price) over semantic tokenization, enabling interpretable and behavior-aligned generation. To address the inefficiency of standard attention, we design a Journey-Aware Sparse Attention (JSA) mechanism, which selectively attends to compressed, intra-, inter-, and current-context segments in the tokenized sequence. Experiments on two real-world datasets show that GRACE significantly outperforms state-of-the-art baselines, achieving up to +106.9% HR@10 and +106.7% NDCG@10 improvement over the state-of-the-art baseline on the Home domain, and +22.1% HR@10 on the Electronics domain. GRACE also reduces attention computation by up to 48% with long sequences.