Compute Requirements for Algorithmic Innovation in Frontier AI Models

TL;DR

This paper empirically analyzes the compute requirements for algorithmic innovations in large language models, revealing that compute caps may not significantly hinder progress due to the resource efficiency of many innovations.

Contribution

It catalogs 36 algorithmic innovations in LLM pretraining, estimates their compute costs, and assesses the impact of compute caps on future innovation.

Findings

Innovations using significant resources double annually.

Compute caps alone are unlikely to drastically slow progress.

Half of the innovations could have been achieved under strict compute limits.

Abstract

Algorithmic innovation in the pretraining of large language models has driven a massive reduction in the total compute required to reach a given level of capability. In this paper we empirically investigate the compute requirements for developing algorithmic innovations. We catalog 36 pre-training algorithmic innovations used in Llama 3 and DeepSeek-V3. For each innovation we estimate both the total FLOP used in development and the FLOP/s of the hardware utilized. Innovations using significant resources double in their requirements each year. We then use this dataset to investigate the effect of compute caps on innovation. Our analysis suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress. Even stringent compute caps -- such as capping total operations to the compute used to train GPT-2 or capping hardware capacity to 8 H100 GPUs -- could still have allowed for half of the cataloged innovations.