Scaling Recommender Transformers to One Billion Parameters

TL;DR

This paper demonstrates how to train and deploy large-scale transformer recommender models with up to one billion parameters, significantly improving recommendation quality in a real-world music platform.

Contribution

It introduces a scalable training recipe for billion-parameter transformer recommenders and shows effective decomposition of autoregressive learning tasks.

Findings

Achieved successful deployment on a large-scale music platform

Online A/B tests show +2.26% increase in total listening time

User liking likelihood increased by +6.37%

Abstract

While large transformer models have been successfully used in many real-world applications such as natural language processing, computer vision, and speech processing, scaling transformers for recommender systems remains a challenging problem. Recently, Generative Recommenders framework was proposed to scale beyond typical Deep Learning Recommendation Models (DLRMs). Reformulation of recommendation as sequential transduction task led to improvement of scaling properties in terms of compute. Nevertheless, the largest encoder configuration reported by the HSTU authors amounts only to ~176 million parameters, which is considerably smaller than the hundreds of billions or even trillions of parameters common in modern language models. In this work, we present a recipe for training large transformer recommenders with up to a billion parameters. We show that autoregressive learning on user histories naturally decomposes into two subtasks, feedback prediction and next-item prediction, and demonstrate that such a decomposition scales effectively across a wide range of transformer sizes. Furthermore, we report a successful deployment of our proposed architecture on a large-scale music platform serving millions of users. According to our online A/B tests, this new model increases total listening time by +2.26% and raises the likelihood of user likes by +6.37%, constituting (to our knowledge) the largest improvement in recommendation quality reported for any deep learning-based system in the platform's history.