Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

TL;DR

This paper explores using large language models as annotators to improve NLP model generalization across domains, proposing a novel sampling strategy that outperforms traditional active learning methods.

Contribution

It introduces a new sampling algorithm based on prediction score differences between base and fine-tuned models for effective annotation with LLMs.

Findings

Significant accuracy improvements in classification and ranking tasks.

Traditional uncertainty sampling is less effective than the proposed method.

The approach reduces annotation costs while enhancing domain generalization.

Abstract

State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of large language models (LLMs) for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.