Scaling Up Active Testing to Large Language Models

TL;DR

This paper presents scalable active testing methods for large language models, reducing computational costs by using in-context learning for surrogate models and introducing a bootstrap estimator for evaluation error.

Contribution

It introduces cost-effective active testing techniques for LLMs, including surrogate models built with in-context learning and a bootstrap estimator for evaluation accuracy.

Findings

Active testing improves LLM evaluation accuracy over random sampling.

Surrogate models can be smaller and built without updating during testing.

Bootstrap estimator effectively indicates active testing performance.

Abstract

Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.