The Price of Progress: Price Performance and the Future of AI

TL;DR

This paper analyzes the decreasing costs of AI model performance over time, revealing rapid price reductions driven by efficiency improvements and highlighting the importance of considering benchmarking costs when assessing AI progress.

Contribution

It provides the largest dataset of AI benchmark prices, quantifies the rate of cost reductions, and emphasizes the need to account for economic factors in evaluating AI advancements.

Findings

AI benchmark costs have decreased 5-10x per year for frontier models.

Algorithmic efficiency improves at about 3x per year after controlling for hardware costs.

Running costs for frontier models are rising 3-18x per year due to larger models.

Abstract

Language models have seen enormous progress on advanced benchmarks in recent years, but much of this progress has only been possible by using more costly models. Benchmarks may therefore present a warped picture of progress in practical capabilities *per dollar*. To remedy this, we use data from Artificial Analysis and Epoch AI to form the largest dataset of current and historical prices to run benchmarks to date. We find that the price for a given level of benchmark performance has decreased remarkably fast, around $5\times$ to $10\times$ per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks. These reductions in the cost of AI inference are due to economic forces, hardware efficiency improvements, and algorithmic efficiency improvements. Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around $3\times$ per year. However, at the same time, the price of running frontier models is rising between $3\times$ to $18\times$ per year due to bigger models and larger reasoning demands. Finally, we recommend that evaluators both publicize and take into account the price of benchmarking as an essential part of measuring the real-world impact of AI.