Impact of Stop Sets on Stopping Active Learning for Text Classification

TL;DR

This paper examines how the choice of stop sets influences the effectiveness of stopping methods in active learning for text classification, revealing significant performance differences and emphasizing the importance of unbiased, representative stop sets.

Contribution

It provides the first comprehensive analysis of stop set choices on stopping method performance, highlighting the superiority of unbiased, representative stop sets and their impact on stability-based methods.

Findings

Unbiased stop sets outperform biased ones in stopping method performance.

Stability-based stopping methods perform better with unbiased stop sets.

Stop set choice significantly affects the practical effectiveness of active learning.

Abstract

Active learning is an increasingly important branch of machine learning and a powerful technique for natural language processing. The main advantage of active learning is its potential to reduce the amount of labeled data needed to learn high-performing models. A vital aspect of an effective active learning algorithm is the determination of when to stop obtaining additional labeled data. Several leading state-of-the-art stopping methods use a stop set to help make this decision. However, there has been relatively less attention given to the choice of stop set than to the stopping algorithms that are applied on the stop set. Different choices of stop sets can lead to significant differences in stopping method performance. We investigate the impact of different stop set choices on different stopping methods. This paper shows the choice of the stop set can have a significant impact on the performance of stopping methods and the impact is different for stability-based methods from that on confidence-based methods. Furthermore, the unbiased representative stop sets suggested by original authors of methods work better than the systematically biased stop sets used in recently published work, and stopping methods based on stabilizing predictions have stronger performance than confidence-based stopping methods when unbiased representative stop sets are used. We provide the largest quantity of experimental results on the impact of stop sets to date. The findings are important for helping to illuminate the impact of this important aspect of stopping methods that has been under-considered in recently published work and that can have a large practical impact on the performance of stopping methods for important semantic computing applications such as technology assisted review and text classification more broadly.