Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design

TL;DR

Kunlun is a scalable architecture for recommendation systems that improves efficiency and resource utilization, enabling predictable scaling laws and significant performance gains in large-scale industrial applications.

Contribution

The paper introduces Kunlun, a unified architecture with novel low- and high-level optimizations that enhance scaling efficiency and model performance for recommendation systems.

Findings

Increased Model FLOPs Utilization from 17% to 37%.

Doubled scaling efficiency compared to previous methods.

Deployed in Meta Ads with significant production impact.

Abstract

Deriving predictable scaling laws that govern the relationship between model performance and computational investment is crucial for designing and allocating resources in massive-scale recommendation systems. While such laws are established for large language models, they remain challenging for recommendation systems, especially those processing both user history and context features. We identify poor scaling efficiency as the main barrier to predictable power-law scaling, stemming from inefficient modules with low Model FLOPs Utilization (MFU) and suboptimal resource allocation. We introduce Kunlun, a scalable architecture that systematically improves model efficiency and resource allocation. Our low-level optimizations include Generalized Dot-Product Attention (GDPA), Hierarchical Seed Pooling (HSP), and Sliding Window Attention. Our high-level innovations feature Computation Skip (CompSkip) and Event-level Personalization. These advances increase MFU from 17% to 37% on NVIDIA B200 GPUs and double scaling efficiency over state-of-the-art methods. Kunlun is now deployed in major Meta Ads models, delivering significant production impact.