Data Efficiency for Large Recommendation Models

TL;DR

This paper provides principles and frameworks to optimize training data efficiency for large recommendation models, reducing costs and improving R&D velocity in high-scale online advertising systems.

Contribution

It introduces data convergence concepts and methods to accelerate convergence, guiding practitioners to balance data volume and model size effectively.

Findings

Strategies successfully deployed in Google's Ads CTR models

Frameworks applicable beyond large recommendation models

Guidelines for balancing data volume and model complexity

Abstract

Large recommendation models (LRMs) are fundamental to the multi-billion dollar online advertising industry, processing massive datasets of hundreds of billions of examples before transitioning to continuous online training to adapt to rapidly changing user behavior. The massive scale of data directly impacts both computational costs and the speed at which new methods can be evaluated (R&D velocity). This paper presents actionable principles and high-level frameworks to guide practitioners in optimizing training data requirements. These strategies have been successfully deployed in Google's largest Ads CTR prediction models and are broadly applicable beyond LRMs. We outline the concept of data convergence, describe methods to accelerate this convergence, and finally, detail how to optimally balance training data volume with model size.