A Novel Two-Step Fine-Tuning Pipeline for Cold-Start Active Learning in Text Classification Tasks

TL;DR

This paper introduces DoTCAL, a two-step fine-tuning pipeline leveraging unlabeled data and active learning to improve BERT-based text classification in cold-start scenarios, showing significant performance gains.

Contribution

The paper proposes a novel two-step fine-tuning pipeline, DoTCAL, that reduces labeled data reliance and enhances active learning effectiveness for BERT in cold-start text classification.

Findings

DoTCAL outperforms traditional methods with up to 33% higher Macro-F1.

BOW and LSI sometimes outperform BERT, especially in low-resource tasks.

Using unlabeled data via domain adaptation improves model performance.

Abstract

This is the first work to investigate the effectiveness of BERT-based contextual embeddings in active learning (AL) tasks on cold-start scenarios, where traditional fine-tuning is infeasible due to the absence of labeled data. Our primary contribution is the proposal of a more robust fine-tuning pipeline - DoTCAL - that diminishes the reliance on labeled data in AL using two steps: (1) fully leveraging unlabeled data through domain adaptation of the embeddings via masked language modeling and (2) further adjusting model weights using labeled data selected by AL. Our evaluation contrasts BERT-based embeddings with other prevalent text representation paradigms, including Bag of Words (BoW), Latent Semantic Indexing (LSI), and FastText, at two critical stages of the AL process: instance selection and classification. Experiments conducted on eight ATC benchmarks with varying AL budgets (number of labeled instances) and number of instances (about 5,000 to 300,000) demonstrate DoTCAL's superior effectiveness, achieving up to a 33% improvement in Macro-F1 while reducing labeling efforts by half compared to the traditional one-step method. We also found that in several tasks, BoW and LSI (due to information aggregation) produce results superior (up to 59% ) to BERT, especially in low-budget scenarios and hard-to-classify tasks, which is quite surprising.