SIDE: Semantic ID Embedding for effective learning from sequences

TL;DR

This paper introduces SIDE, a novel semantic ID embedding method using vector quantization to improve large-scale sequence-based recommendation systems by reducing storage and inference costs.

Contribution

The paper presents a multi-task VQ-VAE framework, a parameter-free SID-to-embedding conversion, and a new quantization method called Discrete-PCA, advancing scalable recommendation models.

Findings

Achieves 2.4X improvement in normalized entropy gain.

Reduces data footprint by 3X compared to traditional SID methods.

Validates effectiveness on a large-scale industrial ads-recommendation system.

Abstract

Sequence-based recommendations models are driving the state-of-the-art for industrial ad-recommendation systems. Such systems typically deal with user histories or sequence lengths ranging in the order of O(10^3) to O(10^4) events. While adding embeddings at this scale is manageable in pre-trained models, incorporating them into real-time prediction models is challenging due to both storage and inference costs. To address this scaling challenge, we propose a novel approach that leverages vector quantization (VQ) to inject a compact Semantic ID (SID) as input to the recommendation models instead of a collection of embeddings. Our method builds on recent works of SIDs by introducing three key innovations: (i) a multi-task VQ-VAE framework, called VQ fusion that fuses multiple content embeddings and categorical predictions into a single Semantic ID; (ii) a parameter-free, highly granular SID-to-embedding conversion technique, called SIDE, that is validated with two content embedding collections, thereby eliminating the need for a large parameterized lookup table; and (iii) a novel quantization method called Discrete-PCA (DPCA) which generalizes and enhances residual quantization techniques. The proposed enhancements when applied to a large-scale industrial ads-recommendation system achieves 2.4X improvement in normalized entropy (NE) gain and 3X reduction in data footprint compared to traditional SID methods.