Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts

TL;DR

This paper introduces Fuxi-MME, a scalable recommendation framework combining multi-embedding and Mixture-of-Experts to better capture item diversity and relevance, outperforming existing models.

Contribution

It proposes a novel framework integrating multi-embedding and MoE architectures to enhance scalability and representation diversity in sequential recommendation models.

Findings

Fuxi-MME outperforms several baseline models on public datasets.

The multi-embedding strategy effectively captures diverse item characteristics.

The MoE layer provides adaptive transformation of item representations.

Abstract

In recommendation systems, how to effectively scale up recommendation models has been an essential research topic. While significant progress has been made in developing advanced and scalable architectures for sequential recommendation(SR) models, there are still challenges due to items' multi-faceted characteristics and dynamic item relevance in the user context. To address these issues, we propose Fuxi-MME, a framework that integrates a multi-embedding strategy with a Mixture-of-Experts (MoE) architecture. Specifically, to efficiently capture diverse item characteristics in a decoupled manner, we decompose the conventional single embedding matrix into several lower-dimensional embedding matrices. Additionally, by substituting relevant parameters in the Fuxi Block with an MoE layer, our model achieves adaptive and specialized transformation of the enriched representations. Empirical results on public datasets show that our proposed framework outperforms several competitive baselines.