Cold-start Active Learning through Self-supervised Language Modeling

TL;DR

This paper introduces a novel cold-start active learning method using self-supervised language modeling loss from pre-trained models like BERT to efficiently select examples for labeling, reducing annotation costs.

Contribution

It proposes a new active learning strategy leveraging language modeling loss as a proxy for uncertainty, effective in cold-start scenarios with limited data.

Findings

Achieves higher accuracy with fewer samples compared to baselines

Reduces labeling costs and computational time

Effective in cold-start text classification tasks

Abstract

Active learning strives to reduce annotation costs by choosing the most critical examples to label. Typically, the active learning strategy is contingent on the classification model. For instance, uncertainty sampling depends on poorly calibrated model confidence scores. In the cold-start setting, active learning is impractical because of model instability and data scarcity. Fortunately, modern NLP provides an additional source of information: pre-trained language models. The pre-training loss can find examples that surprise the model and should be labeled for efficient fine-tuning. Therefore, we treat the language modeling loss as a proxy for classification uncertainty. With BERT, we develop a simple strategy based on the masked language modeling loss that minimizes labeling costs for text classification. Compared to other baselines, our approach reaches higher accuracy within less sampling iterations and computation time.