Scaling Transformers for Discriminative Recommendation via Generative Pretraining

TL;DR

This paper introduces GPSD, a framework that uses generative pretraining and sparse parameter freezing to improve the scalability and generalization of large discriminative recommendation models, effectively addressing overfitting issues.

Contribution

GPSD is a novel framework that leverages generative pretraining to initialize discriminative models and employs sparse parameter freezing, enhancing scalability and reducing overfitting in recommendation systems.

Findings

GPSD significantly narrows the generalization gap in model training.

It achieves consistent performance improvements as model size scales from 13K to 0.3B parameters.

GPSD delivers superior online A/B test results in industrial settings.

Abstract

Discriminative recommendation tasks, such as CTR (click-through rate) and CVR (conversion rate) prediction, play critical roles in the ranking stage of large-scale industrial recommender systems. However, training a discriminative model encounters a significant overfitting issue induced by data sparsity. Moreover, this overfitting issue worsens with larger models, causing them to underperform smaller ones. To address the overfitting issue and enhance model scalability, we propose a framework named GPSD (\textbf{G}enerative \textbf{P}retraining for \textbf{S}calable \textbf{D}iscriminative Recommendation), drawing inspiration from generative training, which exhibits no evident signs of overfitting. GPSD leverages the parameters learned from a pretrained generative model to initialize a discriminative model, and subsequently applies a sparse parameter freezing strategy. Extensive experiments conducted on both industrial-scale and publicly available datasets demonstrate the superior performance of GPSD. Moreover, it delivers remarkable improvements in online A/B tests. GPSD offers two primary advantages: 1) it substantially narrows the generalization gap in model training, resulting in better test performance; and 2) it leverages the scalability of Transformers, delivering consistent performance gains as models are scaled up. Specifically, we observe consistent performance improvements as the model dense parameters scale from 13K to 0.3B, closely adhering to power laws. These findings pave the way for unifying the architectures of recommendation models and language models, enabling the direct application of techniques well-established in large language models to recommendation models. The code is available at https://github.com/chqiwang/gpsd-rec.