FuXi-$\gamma$: Efficient Sequential Recommendation with Exponential-Power Temporal Encoder and Diagonal-Sparse Positional Mechanism

TL;DR

FuXi-$fectively improves sequential recommendation by introducing an exponential-power temporal encoder and a diagonal-sparse positional mechanism, achieving state-of-the-art results with significantly enhanced efficiency and scalability.

Contribution

It presents FuXi-$fectively combines a novel temporal encoding and sparse attention mechanism within a Transformer framework for improved recommendation performance.

Findings

Achieves state-of-the-art recommendation accuracy on four datasets.

Speeds up training by up to 4.74 times and inference by up to 6.18 times.

Effectively models both short-term and long-term user preferences.

Abstract

Sequential recommendation aims to model users' evolving preferences based on their historical interactions. Recent advances leverage Transformer-based architectures to capture global dependencies, but existing methods often suffer from high computational overhead, primarily due to discontinuous memory access in temporal encoding and dense attention over long sequences. To address these limitations, we propose FuXi-$\gamma$, a novel sequential recommendation framework that improves both effectiveness and efficiency through principled architectural design. FuXi-$\gamma$ adopts a decoder-only Transformer structure and introduces two key innovations: (1) An exponential-power temporal encoder that encodes relative temporal intervals using a tunable exponential decay function inspired by the Ebbinghaus forgetting curve. This encoder enables flexible modeling of both short-term and long-term preferences while maintaining high efficiency through continuous memory access and pure matrix operations. (2) A diagonal-sparse positional mechanism that prunes low-contribution attention blocks using a diagonal-sliding strategy guided by the persymmetry of Toeplitz matrix. Extensive experiments on four real-world datasets demonstrate that FuXi-$\gamma$ achieves state-of-the-art performance in recommendation quality, while accelerating training by up to 4.74$\times$ and inference by up to 6.18$\times$, making it a practical and scalable solution for long-sequence recommendation. Our code is available at https://github.com/Yeedzhi/FuXi-gamma.