Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking

TL;DR

This paper introduces SIREN, a new evaluation protocol for large language models that corrects biases in adaptive benchmarking, providing more reliable performance estimates within fixed budgets.

Contribution

The paper proposes SIREN, a selection-aware reporting protocol that improves the reliability of LLM evaluation by separating selection from evaluation and quantifying uncertainty.

Findings

SIREN provides valid confidence intervals for procedure performance.

Winner-based reporting can be overly optimistic and misleading.

SIREN closely matches finite-sample reporting targets in experiments.

Abstract

Adaptive prompt and program search makes LLM evaluation selection-sensitive. Once benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedure-performance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target.