How Benchmark Prediction from Fewer Data Misses the Mark

TL;DR

This paper critically evaluates 11 benchmark prediction methods for LLM evaluation, revealing their dependence on model similarity and introducing a new method that modestly improves extrapolation performance, yet highlighting fundamental limitations at the evaluation frontier.

Contribution

It systematically assesses existing benchmark prediction methods, identifies a strong simple baseline, and proposes a new weighting method to improve extrapolation, exposing key limitations.

Findings

Random sampling with regression is a strong baseline.

Existing methods depend heavily on model similarity.

Benchmark prediction struggles with extrapolating to new, more capable models.

Abstract

Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.