MSN: A Memory-based Sparse Activation Scaling Framework for Large-scale Industrial Recommendation

TL;DR

MSN introduces a memory-based sparse activation framework for large-scale recommendation systems, enhancing personalization and efficiency by dynamically retrieving representations from a large memory while controlling computational costs.

Contribution

The paper proposes MSN, a novel framework that integrates large parameterized memory with sparse activation and efficient retrieval mechanisms for improved industrial recommendation models.

Findings

MSN improves recommendation accuracy over state-of-the-art models.

MSN maintains low computational overhead suitable for industrial deployment.

MSN achieves significant online performance gains in Douyin Search Ranking.

Abstract

Scaling deep learning recommendation models is an effective way to improve model expressiveness. Existing approaches often incur substantial computational overhead, making them difficult to deploy in large-scale industrial systems under strict latency constraints. Recent sparse activation scaling methods, such as Sparse Mixture-of-Experts, reduce computation by activating only a subset of parameters, but still suffer from high memory access costs and limited personalization capacity due to the large size and small number of experts. To address these challenges, we propose MSN, a memory-based sparse activation scaling framework for recommendation models. MSN dynamically retrieves personalized representations from a large parameterized memory and integrates them into downstream feature interaction modules via a memory gating mechanism, enabling fine-grained personalization with low computational overhead. To enable further expansion of the memory capacity while keeping both computational and memory access costs under control, MSN adopts a Product-Key Memory (PKM) mechanism, which factorizes the memory retrieval complexity from linear time to sub-linear complexity. In addition, normalization and over-parameterization techniques are introduced to maintain balanced memory utilization and prevent memory retrieval collapse. We further design customized Sparse-Gather operator and adopt the AirTopK operator to improve training and inference efficiency in industrial settings. Extensive experiments demonstrate that MSN consistently improves recommendation performance while maintaining high efficiency. Moreover, MSN has been successfully deployed in the Douyin Search Ranking System, achieving significant gains over deployed state-of-the-art models in both offline evaluation metrics and large-scale online A/B test.