On the Origin of Algorithmic Progress in AI

TL;DR

This paper investigates the origins of AI algorithmic progress from 2012 to 2023, revealing that much of the efficiency gains are due to scale-dependent effects rather than new innovations, challenging previous assumptions.

Contribution

The study quantifies the contribution of scale-dependent efficiency improvements to AI progress and highlights the dominant role of the LSTM-to-Transformer transition in recent gains.

Findings

Less than 10x of the 22,000x efficiency gains are explained by known innovations.

Scale-dependent effects account for most of the remaining efficiency improvements.

The LSTM-to-Transformer transition explains the majority of the 6,930x total efficiency gains.

Abstract

Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 [Ho et al., 2024]. Running small-scale ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains. Surveying the broader literature, we estimate that additional innovations not included in our ablations account for less than 10x, yielding a total under 100x. This leads us to conduct scaling experiments, which reveal that much of this efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that - contrary to standard assumptions - an algorithm's efficiency gains are tied to compute scale. Using experimental extrapolation and literature estimates, we account for 6,930x efficiency gains over the same time period, with the scale-dependent LSTM-to-Transformer transition accounting for the majority of gains. Our results indicate that algorithmic progress for small models has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.