Your Causal Self-Attentive Recommender Hosts a Lonely Neighborhood
Yueqi Wang, Zhankui He, Zhenrui Yue, Julian McAuley, Dong Wang

TL;DR
This paper provides a theoretical and empirical comparison of auto-encoding and auto-regressive self-attention mechanisms in sequential recommendation, revealing AR's advantages in sparse neighborhood effects and offering a modular experimental pipeline for future research.
Contribution
It offers the first comprehensive theoretical analysis of AE/AR attention matrices and introduces ModSAR, a modular pipeline for empirical evaluation and model development in self-attentive recommenders.
Findings
AR attention exhibits sparse neighborhood effects suitable for sparse scenarios
Empirical results show AR performs better overall on benchmarks
ModSAR facilitates modular and adaptive experimentation
Abstract
In the context of sequential recommendation, a pivotal issue pertains to the comparative analysis between bi-directional/auto-encoding (AE) and uni-directional/auto-regressive (AR) attention mechanisms, where the conclusions regarding architectural and performance superiority remain inconclusive. Previous efforts in such comparisons primarily involve summarizing existing works to identify a consensus or conducting ablation studies on peripheral modeling techniques, such as choices of loss functions. However, far fewer efforts have been made in (1) theoretical and (2) extensive empirical analysis of the self-attention module, the very pivotal structure on which performance and designing insights should be anchored. In this work, we first provide a comprehensive theoretical analysis of AE/AR attention matrix in the aspect of (1) sparse local inductive bias, a.k.a neighborhood effects, and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Causal Inference Techniques · Decision-Making and Behavioral Economics · Mental Health Research Topics
MethodsAutoencoders
