SPARC: Soft Probabilistic Adaptive multi-interest Retrieval Model via Codebooks for recommender system

TL;DR

SPARC introduces a dynamic, probabilistic multi-interest retrieval model using codebooks and RQ-VAE, enabling interest evolution and proactive exploration, significantly improving recommendation performance.

Contribution

The paper presents a novel framework combining RQ-VAE and probabilistic interest modeling to dynamically evolve user interests and enable proactive interest discovery in recommender systems.

Findings

Online platform gains: +0.9% user view duration, +0.4% page views, +22.7% PV500.

Offline metrics: Improved Recall@K and NDCG@K scores.

Validated effectiveness through large-scale online and offline experiments.

Abstract

Modeling multi-interests has arisen as a core problem in real-world RS. Current multi-interest retrieval methods pose three major challenges: 1) Interests, typically extracted from predefined external knowledge, are invariant. Failed to dynamically evolve with users' real-time consumption preferences. 2) Online inference typically employs an over-exploited strategy, mainly matching users' existing interests, lacking proactive exploration and discovery of novel and long-tail interests. To address these challenges, we propose a novel retrieval framework named SPARC(Soft Probabilistic Adaptive Retrieval Model via Codebooks). Our contribution is two folds. First, the framework utilizes Residual Quantized Variational Autoencoder (RQ-VAE) to construct a discretized interest space. It achieves joint training of the RQ-VAE with the industrial large scale recommendation model, mining behavior-aware interests that can perceive user feedback and evolve dynamically. Secondly, a probabilistic interest module that predicts the probability distribution over the entire dynamic and discrete interest space. This facilitates an efficient "soft-search" strategy during online inference, revolutionizing the retrieval paradigm from "passive matching" to "proactive exploration" and thereby effectively promoting interest discovery. Online A/B tests on an industrial platform with tens of millions daily active users, have achieved substantial gains in business metrics: +0.9% increase in user view duration, +0.4% increase in user page views (PV), and a +22.7% improvement in PV500(new content reaching 500 PVs in 24 hours). Offline evaluations are conducted on open-source Amazon Product datasets. Metrics, such as Recall@K and Normalized Discounted Cumulative Gain@K(NDCG@K), also showed consistent improvement. Both online and offline experiments validate the efficacy and practical value of the proposed method.