Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

TL;DR

This paper introduces two novel causal attention architectures for generative recommender systems that improve efficiency and performance by explicitly modeling item-action dependencies, reducing sequence complexity and training time.

Contribution

It proposes two new architectures, AttnLFA and AttnMVP, that eliminate interleaved dependencies, explicitly encode causality, and outperform traditional interleaving methods in large-scale recommendation tasks.

Findings

AttnLFA and AttnMVP outperform interleaved baselines in loss and entropy.

Models reduce sequence complexity by 50%.

Training time decreases by up to 23%.

Abstract

Generative Recommender Systems (GR) increasingly model user behavior as a sequence generation task by interleaving item and action tokens. While effective, this formulation introduces significant structural and computational inefficiencies: it doubles sequence length, incurs quadratic overhead, and relies on implicit attention to recover the causal relationship between an item and its associated action. Furthermore, interleaving heterogeneous tokens forces the Transformer to disentangle semantically incompatible signals, leading to increased attention noise and reduced representation efficiency.In this work, we propose a principled reformulation of generative recommendation that aligns sequence modeling with underlying causal structures and attention theory. We demonstrate that current interleaving mechanisms act as inefficient proxies for similarity-weighted action pooling. To address this, we introduce two novel architectures that eliminate interleaved dependencies to reduce sequence complexity by 50%: Attention-based Late Fusion for Actions (AttnLFA) and Attention-based Mixed Value Pooling (AttnMVP). These models explicitly encode the $i_n \rightarrow a_n$ causal dependency while preserving the expressive power of Transformer-based sequence modeling.We evaluate our framework on large-scale product recommendation data from a major social network. Experimental results show that AttnLFA and AttnMVP consistently outperform interleaved baselines, achieving evaluation loss improvements of 0.29% and 0.80%, and significant gains in Normalized Entropy (NE). Crucially, these performance gains are accompanied by training time reductions of 23% and 12%, respectively. Our findings suggest that explicitly modeling item-action causality provides a superior design paradigm for scalable and efficient generative ranking.