The Race to Efficiency: A New Perspective on AI Scaling Laws

TL;DR

This paper introduces a new framework called the relative-loss equation that incorporates efficiency and time into AI scaling laws, offering a roadmap for sustainable AI progress amid rising costs.

Contribution

It extends classical scaling laws by integrating efficiency and time, providing a quantitative model to balance hardware investments with algorithmic improvements.

Findings

Efficiency gains can significantly accelerate AI scaling.

Without efficiency improvements, progress could take millennia.

Sustained efficiency doubling rates can enable exponential AI growth.

Abstract

As large-scale AI models expand, training becomes costlier and sustaining progress grows harder. Classical scaling laws (e.g., Kaplan et al. (2020), Hoffmann et al. (2022)) predict training loss from a static compute budget yet neglect time and efficiency, prompting the question: how can we balance ballooning GPU fleets with rapidly improving hardware and algorithms? We introduce the relative-loss equation, a time- and efficiency-aware framework that extends classical AI scaling laws. Our model shows that, without ongoing efficiency gains, advanced performance could demand millennia of training or unrealistically large GPU fleets. However, near-exponential progress remains achievable if the "efficiency-doubling rate" parallels Moore's Law. By formalizing this race to efficiency, we offer a quantitative roadmap for balancing front-loaded GPU investments with incremental improvements across the AI stack. Empirical trends suggest that sustained efficiency gains can push AI scaling well into the coming decade, providing a new perspective on the diminishing returns inherent in classical scaling.