# State-of-the-Art: The Temporal Order of Benchmarking Culture

## TL;DR

This paper explores how benchmarking in machine learning shapes research through time, focusing on its present-oriented and normalizing effects.

## Contribution

It introduces two new hypotheses—normalizing research and extrapolation—to explain the temporal dynamics of benchmarking culture.

## Key findings

- Benchmarking disciplines and motivates research while minimizing conflict.
- The incremental progress in benchmarking is more about maintaining the current state-of-the-art than future advancement.
- The paper identifies a presentist temporality in how machine learning benchmarks are evaluated.

## Abstract

This commentary situates the epistemic values of machine learning’s culture of benchmarking and evaluation within larger temporal structures. Beyond questions of validity, whether model comparisons are statistically valid or whether benchmarks adequately represent meaningful tasks or capabilities, it asks how benchmarks produce certain temporal values and expectations. It articulates two hypotheses in response: the first, termed normalizing research, seeks to characterize how benchmarking simultaneously serves a disciplining and motivating function in research, with the effect of minimizing conflict. The second, termed extrapolation, argues that the incremental, progressive rhythm of benchmarking is oriented less towards the future than towards a present state-of-the-art (SOTA). Together, these hypotheses inform a diagnosis of the presentist temporality of benchmarking and evaluation in machine learning.