Long-Term Embeddings for Balanced Personalization

TL;DR

This paper introduces Long-Term Embeddings (LTE) to improve personalization in transformer recommenders by capturing stable preferences and ensuring cross-version compatibility, addressing recency bias and production deployment challenges.

Contribution

The LTE framework provides a stable, content-based embedding system that maintains cross-version consistency and enhances long-term personalization in recommender systems.

Findings

LTE improves user engagement and financial metrics in online tests.

The proposed methods effectively address the offline-online mismatch issue.

LTE enables behavioral fine-tuning while maintaining stability.

Abstract

Modern transformer-based sequential recommenders excel at capturing short-term intent but often suffer from recency bias, overlooking stable long-term preferences. While extending sequence lengths is an intuitive fix, it is computationally inefficient, and recent interactions tend to dominate the model's attention. We propose Long-Term Embeddings (LTE) as a high-inertia contextual anchor to bridge this gap. We address a critical production challenge: the point-in-time consistency problem caused by infrastructure constraints, as feature stores typically host only a single "live" version of features. This leads to an offline-online mismatch during model deployments and rollbacks, as models are forced to process evolved representations they never saw during training. To resolve this, we introduce an LTE framework that constrains embeddings to a fixed semantic basis of content-based item representations, ensuring cross-version compatibility. Furthermore, we investigate integration strategies for causal language modeling, considering the data leakage issue that occurs when the LTE and the transformer's short-term sequence share a temporal horizon. We evaluate two representations: a heuristic average and an asymmetric autoencoder with a fixed decoder grounded in the semantic basis to enable behavioral fine-tuning while maintaining stability. Online A/B tests on Zalando demonstrate that integrating LTE as a contextual prefix token using a lagged window yields significant uplifts in both user engagement and financial metrics.