From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction

TL;DR

This paper introduces the Field-Aware Transformer (FAT), a novel model for CTR prediction that aligns model structure with data semantics, leading to better scaling, improved accuracy, and a formal understanding of model capacity.

Contribution

We propose FAT, a transformer variant embedding field-based priors, and establish a formal scaling law for CTR models based on Rademacher complexity.

Findings

FAT improves AUC by up to +0.51% on benchmarks.

FAT delivers +2.33% CTR and +0.66% RPM online.

Power-law scaling observed in model performance as width increases.

Abstract

Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns - a stark contrast to the smooth, predictable gains seen in large language models. We identify the root cause as a structural misalignment: Transformers assume sequential compositionality, while CTR data demand combinatorial reasoning over high-cardinality semantic fields. Unstructured attention spreads capacity indiscriminately, amplifying noise under extreme sparsity and breaking scalable learning. To restore alignment, we introduce the Field-Aware Transformer (FAT), which embeds field-based interaction priors into attention through decomposed content alignment and cross-field modulation. This design ensures model complexity scales with the number of fields F, not the total vocabulary size n >> F, leading to tighter generalization and, critically, observed power-law scaling in AUC as model width increases. We present the first formal scaling law for CTR models, grounded in Rademacher complexity, that explains and predicts this behavior. On large-scale benchmarks, FAT improves AUC by up to +0.51% over state-of-the-art methods. Deployed online, it delivers +2.33% CTR and +0.66% RPM. Our work establishes that effective scaling in recommendation arises not from size, but from structured expressivity-architectural coherence with data semantics.