Progress over Points: Reframing LM Benchmarks Around Scientific Objectives

TL;DR

This paper proposes a new, progress-oriented benchmark environment for language models that focuses on scientific objectives, exemplified by a NanoGPT speedrun environment, to better measure meaningful advancements in the field.

Contribution

It introduces a novel benchmark framework centered on scientific progress, standardizes evaluation tools, and demonstrates improved training efficiency and emergent algorithms.

Findings

Achieved a new state-of-the-art training time, reducing it by 3 seconds.

Observed emergence of novel algorithmic ideas during benchmarking.

Promotes a shift from static leaderboards to scientific progress measurement.

Abstract

Current benchmarks that test LLMs on static, already-solved problems (e.g., math word problems) effectively demonstrated basic capability acquisition. The natural progression has been toward larger, more comprehensive and challenging collections of static problems, an approach that inadvertently constrains the kinds of advances we can measure and incentivize. To address this limitation, we argue for progress-oriented benchmarks, problem environments whose objectives are themselves the core targets of scientific progress, so that achieving state of the art on the benchmark advances the field. As a introductory step, we instantiate an environment based on the NanoGPT speedrun. The environment standardizes a dataset slice, a reference model and training harness, and rich telemetry, with run-time verification and anti-gaming checks. Evaluation centers on the scientific delta achieved: best-attained loss and the efficiency frontier. Using this environment, we achieve a new state-of-the-art training time, improving upon the previous record by 3 seconds, and qualitatively observe the emergence of novel algorithmic ideas. Moreover, comparisons between models and agents remain possible, but they are a means, not the end; the benchmark's purpose is to catalyze reusable improvements to the language modeling stack. With this release, the overarching goal is to seed a community shift from static problem leaderboards to test-time research on open-ended yet measurable scientific problems. In this new paradigm, progress on the benchmark is progress on the science, thus reframing "benchmarking" as a vehicle for scientific advancement.