ALScope: A Unified Toolkit for Deep Active Learning

TL;DR

ALSCOPE is a comprehensive platform that unifies the evaluation of deep active learning algorithms across diverse datasets and challenging scenarios, revealing significant performance variability and areas for improvement.

Contribution

The paper introduces ALScope, a unified toolkit for systematic evaluation of deep active learning algorithms across multiple datasets and challenging conditions.

Findings

DAL performance varies across domains and settings.

Imbalanced and open-set scenarios need further research.

Some algorithms are computationally intensive despite good performance.

Abstract

Deep Active Learning (DAL) reduces annotation costs by selecting the most informative unlabeled samples during training. As real-world applications become more complex, challenges stemming from distribution shifts (e.g., open-set recognition) and data imbalance have gained increasing attention, prompting the development of numerous DAL algorithms. However, the lack of a unified platform has hindered fair and systematic evaluation under diverse conditions. Therefore, we present a new DAL platform ALScope for classification tasks, integrating 10 datasets from computer vision (CV) and natural language processing (NLP), and 21 representative DAL algorithms, including both classical baselines and recent approaches designed to handle challenges such as distribution shifts and data imbalance. This platform supports flexible configuration of key experimental factors, ranging from algorithm and dataset choices to task-specific factors like out-of-distribution (OOD) sample ratio, and class imbalance ratio, enabling comprehensive and realistic evaluation. We conduct extensive experiments on this platform under various settings. Our findings show that: (1) DAL algorithms' performance varies significantly across domains and task settings; (2) in non-standard scenarios such as imbalanced and open-set settings, DAL algorithms show room for improvement and require further investigation; and (3) some algorithms achieve good performance, but require significantly longer selection time.