Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings

TL;DR

This paper introduces Scales++, an item-centric method for selecting small, representative benchmark subsets based on task item properties, significantly reducing evaluation costs while maintaining accuracy.

Contribution

It proposes a novel item-centric approach for benchmark subset selection, focusing on task item properties rather than model failure patterns, with a practical implementation called Scales++.

Findings

Scales++ reduces selection cost by over 18x.

Predicts full benchmark scores with 3.2% MAE using 0.25% data.

Achieves 2.9% MAE on Humanity's Last Exam with 2.0% data.

Abstract

The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative data subsets (i.e., tiny benchmarks) that enable efficient assessment while retaining predictive fidelity. Current methods for this task operate under a model-centric paradigm, selecting benchmarking items based on the collective performance of existing models. Such approaches are limited by large upfront costs, an inability to immediately handle new benchmarks ("cold-start"), and the fragile assumption that future models will share the failure patterns of their predecessors. In this work, we propose a new item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves, rather than on model-specific failure patterns. We instantiate this item-centric efficient benchmarking approach via a novel method, Scales++, where data selection is based on the cognitive demands of the benchmark samples. Empirically, we show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity. On the Open LLM Leaderboard, using just a 0.25% data subset, we predict full benchmark scores with a 3.2% mean absolute error, and on Humanity's Last Exam we predict full scores with 2.9% mean absolute error using a 2.0% sample. We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation, while also providing better cold-start performance and more interpretable benchmarking.