Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs

TL;DR

Cer-Eval is a novel evaluation framework for large language models that adaptively selects test samples to reduce evaluation costs by 20-40% while ensuring high-confidence performance estimates.

Contribution

It introduces a certifiable, cost-efficient evaluation method with theoretical bounds and an adaptive algorithm for test sample selection in LLM assessment.

Findings

Reduces test sample requirements by 20-40%.

Maintains high-confidence evaluation accuracy.

Provides theoretical bounds on test sample complexity.

Abstract

As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.