LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation

TL;DR

This paper introduces LLaTTE, a scalable transformer architecture for ads recommendation that leverages power-law scaling, semantic features, and a two-stage model to improve performance while maintaining low latency in industrial settings.

Contribution

It demonstrates that recommendation sequence modeling follows predictable scaling laws and introduces a two-stage architecture to effectively utilize large models under latency constraints.

Findings

Scaling laws apply to recommendation sequence modeling.

Semantic features are essential for effective scaling.

The multi-stage model achieves a 4.3% conversion uplift at Meta.

Abstract

We present LLaTTE (LLM-Style Latent Transformers for Temporal Events), a scalable transformer architecture for production ads recommendation. Through systematic experiments, we demonstrate that sequence modeling in recommendation systems follows predictable power-law scaling similar to LLMs. Crucially, we find that semantic features bend the scaling curve: they are a prerequisite for scaling, enabling the model to effectively utilize the capacity of deeper and longer architectures. To realize the benefits of continued scaling under strict latency constraints, we introduce a two-stage architecture that offloads the heavy computation of large, long-context models to an asynchronous upstream user model. We demonstrate that upstream improvements transfer predictably to downstream ranking tasks. Deployed as the largest user model at Meta, this multi-stage framework drives a 4.3\% conversion uplift on Facebook Feed and Reels with minimal serving overhead, establishing a practical blueprint for harnessing scaling laws in industrial recommender systems.