Select, Label, Evaluate: Active Testing in NLP

TL;DR

This paper formalizes Active Testing in NLP, benchmarking various methods across multiple datasets and tasks, demonstrating significant annotation cost reductions while maintaining accurate performance estimates, and introduces an adaptive stopping criterion.

Contribution

It provides a formal framework for Active Testing in NLP, extensive benchmarking of existing methods, and proposes an adaptive stopping criterion to optimize annotation efforts.

Findings

Annotation reductions of up to 95% achieved.

Performance estimates within 1% of full test set.

No single method is universally best across datasets.

Abstract

Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superior. Lastly, to address the limitation of requiring a predefined annotation budget in existing sample selection strategies, we introduce an adaptive stopping criterion that automatically determines the optimal number of samples.