QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks

TL;DR

QuickScope is a new method that efficiently identifies challenging questions in dynamic LLM benchmarks, improving detection accuracy and reducing false positives.

Contribution

It adapts Bayesian optimization for practical LLM evaluation, enabling targeted, sample-efficient discovery of hard questions in dynamic benchmarks.

Findings

QuickScope outperforms standard baselines in discovering difficult questions.

It reduces false positives caused by noisy outcomes.

The method is flexible across various datasets and utility functions.

Abstract

LLM benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive -- especially when the goal is not just determining an average score, but reliably identifying a model's weak spots. This paper introduces a new methodology for identifying hard questions in dynamic benchmarks. It leverages COUP, a recent Bayesian optimization algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of benchmarks, we show that our method, dubbed $\texttt{QuickScope}$, discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes.