The Serial Scaling Hypothesis

TL;DR

This paper explores the limitations of parallel computing in solving inherently serial problems, formalizes this in complexity theory, and shows diffusion models cannot address such tasks effectively.

Contribution

It formalizes the concept of inherently serial problems in complexity theory and demonstrates the limitations of diffusion models in solving them.

Findings

Inherently serial problems cannot be efficiently parallelized.

Diffusion models are incapable of solving inherently serial problems.

Recognizing serial computation has implications for ML model and hardware design.

Abstract

While machine learning has advanced through massive parallelization, we identify a critical blind spot: some problems are fundamentally sequential. These "inherently serial" problems-from mathematical reasoning to physical simulations to sequential decision-making-require sequentially dependent computational steps that cannot be efficiently parallelized. We formalize this distinction in complexity theory, and demonstrate that current parallel-centric architectures face fundamental limitations on such tasks. Then, we show for first time that diffusion models despite their sequential nature are incapable of solving inherently serial problems. We argue that recognizing the serial nature of computation holds profound implications on machine learning, model design, and hardware development.