Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation

TL;DR

This paper introduces DIRECT, an active learning algorithm that effectively handles class imbalance and label noise by identifying class boundaries and selecting uncertain examples, significantly reducing annotation costs.

Contribution

The paper presents the first comprehensive study of active learning under both class imbalance and label noise, with a novel boundary-based selection method called DIRECT.

Findings

Reduces annotation costs by over 60% compared to state-of-the-art methods.

Maintains robustness to label noise in imbalanced datasets.

Achieves over 80% reduction in annotation costs versus random sampling.

Abstract

Class imbalance severely impacts machine learning performance on minority classes in real-world applications. While various solutions exist, active learning offers a fundamental fix by strategically collecting balanced, informative labeled examples from abundant unlabeled data. We introduce DIRECT, an algorithm that identifies class separation boundaries and selects the most uncertain nearby examples for annotation. By reducing the problem to one-dimensional active learning, DIRECT leverages established theory to handle batch labeling and label noise -- another common challenge in data annotation that particularly affects active learning methods. Our work presents the first comprehensive study of active learning under both class imbalance and label noise. Extensive experiments on imbalanced datasets show DIRECT reduces annotation costs by over 60\% compared to state-of-the-art active learning methods and over 80\% versus random sampling, while maintaining robustness to label noise.