Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

TL;DR

Generative Active Testing (GAT) introduces an uncertainty-aware framework that uses LLMs as surrogates and a novel statement adaptation module to efficiently select samples for benchmarking, reducing labeling costs and estimation errors.

Contribution

The paper proposes GAT, a new framework that adapts generative tasks into pseudo-classification formats for better sample selection in LLM evaluation.

Findings

Zero-shot acquisition functions reduce estimation error by ~40%.

GAT enables scalable, cost-effective benchmarking of LLMs.

The Statement Adaptation Module improves uncertainty estimation in generative tasks.

Abstract

With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.