A Rosetta Stone for AI Benchmarks

TL;DR

This paper introduces a statistical framework that unifies AI benchmarks into a single scale, enabling long-term comparison of model capabilities and progress despite benchmark saturation.

Contribution

It presents a novel method to compare AI models across different benchmarks and timeframes without assuming capability evolution patterns.

Findings

Measured AI progress speed over time

Forecasted future AI capabilities

Detected rapid accelerations in AI development

Abstract

Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.