Orthogonal Low Rank Embedding Stabilization

TL;DR

This paper proposes an orthogonal low-rank transformation method to stabilize user and item embeddings across retraining cycles, improving consistency without affecting model performance.

Contribution

It introduces a novel, efficient, and lossless stabilization technique using low-rank SVD and orthogonal Procrustes transformation, preserving embedding quality.

Findings

Ensures consistent embedding spaces across retraining sessions.

Maintains inference quality while reducing operational complexity.

Seamlessly integrates with existing stabilization methods.

Abstract

The instability of embedding spaces across model retraining cycles presents significant challenges to downstream applications using user or item embeddings derived from recommendation systems as input features. This paper introduces a novel orthogonal low-rank transformation methodology designed to stabilize the user/item embedding space, ensuring consistent embedding dimensions across retraining sessions. Our approach leverages a combination of efficient low-rank singular value decomposition and orthogonal Procrustes transformation to map embeddings into a standardized space. This transformation is computationally efficient, lossless, and lightweight, preserving the dot product and inference quality while reducing operational burdens. Unlike existing methods that modify training objectives or embedding structures, our approach maintains the integrity of the primary model application and can be seamlessly integrated with other stabilization techniques.