Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation

TL;DR

The paper introduces MoS, a theme-aware mixture-of-experts framework for long-sequence recommendation that effectively filters irrelevant information and captures multi-scale user behaviors, achieving state-of-the-art results.

Contribution

It proposes a novel MoE-based model with theme-aware routing and multi-scale fusion to improve long-sequence recommendations by handling interest shifts and noisy data.

Findings

MoS outperforms existing models with state-of-the-art accuracy.

MoS achieves this with fewer FLOPs, indicating efficiency.

The framework effectively filters irrelevant information in long sequences.

Abstract

Sequential recommendation has rapidly advanced in click-through rate prediction due to its ability to model dynamic user interests. A key challenge, however, lies in modeling long sequences: users often exhibit significant interest shifts, introducing substantial irrelevant or misleading information. Our empirical analysis corroborates this challenge and uncovers a recurring behavioral pattern in long sequences (\textit{session hopping}): user interests remain stable within short temporal spans (\textit{sessions}) but shift drastically across sessions and may reappear after multiple sessions. To address this challenge, we propose the Mixture of Sequence (MoS) framework, a model-agnostic MoE approach that achieves accurate predictions by extracting theme-specific and multi-scale subsequences from noisy raw user sequences. First, MoS employs a theme-aware routing mechanism to adaptively learn the latent themes of user sequences and organizes these sequences into multiple coherent subsequences. Each subsequence contains only sessions aligned with a specific theme, thereby effectively filtering out irrelevant or even misleading information introduced by user interest shifts in session hopping. In addition, to alleviate potential information loss, we introduce a multi-scale fusion mechanism, which leverages three types of experts to capture global sequence characteristics, short-term user behaviors, and theme-specific semantic patterns. Together, these two mechanisms endow MoS with the ability to deliver accurate recommendations from multi-faceted and multi-scale perspectives. Experimental results demonstrate that MoS consistently achieves the SOTA performance while introducing fewer FLOPs compared with other MoE counterparts, providing strong evidence of its excellent balance between utility and efficiency. The code is available at https://github.com/xiaolin-cs/MoS.