Active learning for reducing labeling effort in text classification tasks

TL;DR

This paper empirically evaluates active learning strategies for text classification using BERT, demonstrating that uncertainty-based AL outperforms random sampling, with performance influenced by query-pool size.

Contribution

It provides the first comprehensive empirical comparison of uncertainty-based active learning algorithms with BERT in text classification tasks.

Findings

Uncertainty-based AL outperforms random sampling with BERT.

Heuristics did not improve AL performance.

Performance gap decreases as query-pool size increases.

Abstract

Labeling data can be an expensive task as it is usually performed manually by domain experts. This is cumbersome for deep learning, as it is dependent on large labeled datasets. Active learning (AL) is a paradigm that aims to reduce labeling effort by only using the data which the used model deems most informative. Little research has been done on AL in a text classification setting and next to none has involved the more recent, state-of-the-art Natural Language Processing (NLP) models. Here, we present an empirical study that compares different uncertainty-based algorithms with BERT$_{base}$ as the used classifier. We evaluate the algorithms on two NLP classification datasets: Stanford Sentiment Treebank and KvK-Frontpages. Additionally, we explore heuristics that aim to solve presupposed problems of uncertainty-based AL; namely, that it is unscalable and that it is prone to selecting outliers. Furthermore, we explore the influence of the query-pool size on the performance of AL. Whereas it was found that the proposed heuristics for AL did not improve performance of AL; our results show that using uncertainty-based AL with BERT$_{base}$ outperforms random sampling of data. This difference in performance can decrease as the query-pool size gets larger.