Efficient Evaluation of LLM Performance with Statistical Guarantees

TL;DR

This paper introduces FAQ, a method for efficiently evaluating large language models by using statistical techniques to reduce the number of queries needed for accurate benchmarking.

Contribution

It presents a novel active querying approach that combines Bayesian modeling and variance reduction to achieve up to 5 times fewer queries while maintaining confidence interval validity.

Findings

FAQ achieves up to 5x sample size gains over baselines.

It maintains valid confidence intervals with fewer queries.

The method is effective across different datasets and missing data levels.

Abstract

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Cand\`es, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.