Effective Frontiers: A Unification of Neural Scaling Laws

TL;DR

This paper introduces a unified theoretical framework for neural scaling laws based on the concept of an Effective Frontier, explaining how model capacity, data size, and compute influence learning through pattern coverage and bottlenecks.

Contribution

It unifies existing neural scaling laws under a single framework using the Effective Frontier concept, providing clear explanations for different scaling regimes.

Findings

Derives precise scaling laws for model capacity, data, and compute.

Shows that different scaling laws are equilibrium solutions of a constrained optimization.

Provides a universal explanation for neural scaling behaviors across architectures.

Abstract

Neural scaling laws govern the prediction power-law improvement of test loss with respect to model capacity ($N$), datasize ($D$), and compute ($C$). However, existing theoretical explanations often rely on specific architectures or complex kernel methods, lacking intuitive universality. In this paper, we propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution. We introduce the Effective Frontier ($k_\star$), a threshold in the pattern rank space that separates learned knowledge from the unlearned tail. We prove that reducible loss is asymptotically determined by the probability mass of the tail a resource-dependent frontier truncation. Based on our framework, we derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks, respectively. Furthermore, we unify these mechanisms via a Max-Bottleneck principle, demonstrating that the Kaplan and Chinchilla scaling laws are not contradictory, but equilibrium solutions to the same constrained optimization problem under different active bottlenecks.