Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learning

TL;DR

REFINE is a novel ensemble active learning method that progressively filters unlabeled data pools by combining multiple strategies, leading to more effective data selection and improved performance across various datasets and models.

Contribution

We introduce REFINE, an ensemble active learning approach that adaptively filters unlabeled pools using multiple strategies, enhancing data selection and outperforming existing methods.

Findings

REFINE outperforms individual strategies and existing ensemble methods across 6 datasets.

Progressive filtering improves the effectiveness of any active learning strategy.

The method enhances performance in an audio spectrogram classification task.

Abstract

Existing active learning (AL) strategies capture fundamentally different notions of data value, e.g., uncertainty or representativeness. Consequently, the effectiveness of strategies can vary substantially across datasets, models, and even AL cycles. Committing to a single strategy risks suboptimal performance, as no single strategy dominates throughout the entire AL process. We introduce REFINE, an ensemble AL method that combines multiple strategies without knowing in advance which will perform best. In each AL cycle, REFINE operates in two stages: (1) Progressive filtering iteratively refines the unlabeled pool by considering an ensemble of AL strategies, retaining promising candidates capturing different notions of value. (2) Coverage-based selection then chooses a final batch from this refined pool, ensuring all previously identified notions of value are accounted for. Extensive experiments across 6 classification datasets and 3 foundation models show that REFINE consistently outperforms individual strategies and existing ensemble methods. Notably, progressive filtering serves as a powerful preprocessing step that improves the performance of any individual AL strategy applied to the refined pool, which we demonstrate on an audio spectrogram classification use case. Finally, the ensemble of REFINE can be easily extended with upcoming state-of-the-art AL strategies.