RQ-GMM: Residual Quantized Gaussian Mixture Model for Multimodal Semantic Discretization in CTR Prediction

TL;DR

This paper introduces RQ-GMM, a probabilistic residual quantization method that improves semantic discretization of multimodal embeddings, leading to better CTR prediction and significant online performance gains.

Contribution

The paper presents RQ-GMM, a novel probabilistic residual quantization approach using Gaussian Mixture Models for improved semantic discretization in CTR prediction.

Findings

Achieves higher codebook utilization and reconstruction accuracy.

Demonstrates a 1.502% increase in Advertiser Value in online A/B tests.

Successfully deployed for large-scale daily recommendations.

Abstract

Multimodal content is crucial for click-through rate (CTR) prediction. However, directly incorporating continuous embeddings from pre-trained models into CTR models yields suboptimal results due to misaligned optimization objectives and convergence speed inconsistency during joint training. Discretizing embeddings into semantic IDs before feeding them into CTR models offers a more effective solution, yet existing methods suffer from limited codebook utilization, reconstruction accuracy, and semantic discriminability. We propose RQ-GMM (Residual Quantized Gaussian Mixture Model), which introduces probabilistic modeling to better capture the statistical structure of multimodal embedding spaces. Through Gaussian Mixture Models combined with residual quantization, RQ-GMM achieves superior codebook utilization and reconstruction accuracy. Experiments on public datasets and online A/B tests on a large-scale short-video platform serving hundreds of millions of users demonstrate substantial improvements: RQ-GMM yields a 1.502% gain in Advertiser Value over strong baselines. The method has been fully deployed, serving daily recommendations for hundreds of millions of users.