Why Generate When You Can Transform? Unleashing Generative Attention for Dynamic Recommendation

TL;DR

This paper introduces generative attention mechanisms for sequential recommendation, leveraging probabilistic models like VAE and Diffusion Models to better capture dynamic user preferences and improve recommendation performance.

Contribution

It presents a theoretical proof of the greater expressiveness of generative attention and introduces two novel models based on VAE and Diffusion Models for SR.

Findings

Models outperform state-of-the-art in accuracy.

Models enhance diversity in recommendations.

Generative attention offers better adaptability.

Abstract

Sequential Recommendation (SR) focuses on personalizing user experiences by predicting future preferences based on historical interactions. Transformer models, with their attention mechanisms, have become the dominant architecture in SR tasks due to their ability to capture dependencies in user behavior sequences. However, traditional attention mechanisms, where attention weights are computed through query-key transformations, are inherently linear and deterministic. This fixed approach limits their ability to account for the dynamic and non-linear nature of user preferences, leading to challenges in capturing evolving interests and subtle behavioral patterns. Given that generative models excel at capturing non-linearity and probabilistic variability, we argue that generating attention distributions offers a more flexible and expressive alternative compared to traditional attention mechanisms. To support this claim, we present a theoretical proof demonstrating that generative attention mechanisms offer greater expressiveness and stochasticity than traditional deterministic approaches. Building upon this theoretical foundation, we introduce two generative attention models for SR, each grounded in the principles of Variational Autoencoders (VAE) and Diffusion Models (DMs), respectively. These models are designed specifically to generate adaptive attention distributions that better align with variable user preferences. Extensive experiments on real-world datasets show our models significantly outperform state-of-the-art in both accuracy and diversity.