Unlocking the Power of Diffusion Models in Sequential Recommendation: A Simple and Effective Approach

TL;DR

This paper introduces ADRec, a novel diffusion-based sequential recommendation framework that addresses embedding collapse by applying token-level diffusion and a three-stage training process, improving recommendation accuracy and efficiency.

Contribution

ADRec is the first to combine token-level diffusion with auto-regression and a multi-stage training strategy to mitigate embedding collapse in sequential recommendation models.

Findings

ADRec outperforms existing methods on six datasets.

The three-stage training improves embedding stability.

Applying denoising only to the last token enhances efficiency.

Abstract

In this paper, we focus on the often-overlooked issue of embedding collapse in existing diffusion-based sequential recommendation models and propose ADRec, an innovative framework designed to mitigate this problem. Diverging from previous diffusion-based methods, ADRec applies an independent noise process to each token and performs diffusion across the entire target sequence during training. ADRec captures token interdependency through auto-regression while modeling per-token distributions through token-level diffusion. This dual approach enables the model to effectively capture both sequence dynamics and item representations, overcoming the limitations of existing methods. To further mitigate embedding collapse, we propose a three-stage training strategy: (1) pre-training the embedding weights, (2) aligning these weights with the ADRec backbone, and (3) fine-tuning the model. During inference, ADRec applies the denoising process only to the last token, ensuring that the meaningful patterns in historical interactions are preserved. Our comprehensive empirical evaluation across six datasets underscores the effectiveness of ADRec in enhancing both the accuracy and efficiency of diffusion-based sequential recommendation systems.