Long-Sequence Recommendation Models Need Decoupled Embeddings

TL;DR

This paper identifies a key limitation in long-sequence recommendation models related to shared embeddings for attention and representation, and proposes a decoupled embedding approach that improves accuracy and efficiency.

Contribution

The paper introduces DARE, a novel model with separate embeddings for attention and representation, addressing a critical deficiency in existing long-sequence recommendation systems.

Findings

DARE outperforms baselines with up to 0.9% AUC improvement on public datasets.

Decoupling embeddings reduces attention embedding dimension and speeds up search by 50%.

Extensive experiments validate the effectiveness of decoupled embeddings in recommendation accuracy and efficiency.

Abstract

Lifelong user behavior sequences are crucial for capturing user interests and predicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: a subset of relevant behaviors is first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections -- a technique borrowed from language processing) proved ineffective, shedding light on the unique challenges of recommendation models. To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accurate searches of correlated behaviors and outperforms baselines with AUC gains up to 0.9% on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50% without significant performance impact, enabling more efficient, high-performance online serving. Code in PyTorch for experiments, including model analysis, is available at https://github.com/thuml/DARE.