Deprecating Benchmarks: Criteria and Framework

TL;DR

This paper proposes criteria and a framework for deprecating AI benchmarks to ensure more accurate and meaningful evaluation of frontier models, addressing issues of over-reliance and safety concerns.

Contribution

It introduces a systematic set of criteria and a structured framework for deprecating benchmarks, improving evaluation quality and governance in AI.

Findings

Established criteria for benchmark deprecation

Developed a framework guiding deprecation processes

Aimed to improve evaluation rigor and safety in AI benchmarking

Abstract

As frontier artificial intelligence (AI) models rapidly advance, benchmarks are integral to comparing different models and measuring their progress in different task-specific domains. However, there is a lack of guidance on when and how benchmarks should be deprecated once they cease to effectively perform their purpose. This risks benchmark scores over-valuing model capabilities, or worse, obscuring capabilities and safety-washing. Based on a review of benchmarking practices, we propose criteria to decide when to fully or partially deprecate benchmarks, and a framework for deprecating benchmarks. Our work aims to advance the state of benchmarking towards rigorous and quality evaluations, especially for frontier models, and our recommendations are aimed to benefit benchmark developers, benchmark users, AI governance actors (across governments, academia, and industry panels), and policy makers.