Active Learning for Open-set Annotation

TL;DR

This paper introduces LfOSA, a novel active learning framework for open-set annotation that effectively detects known class examples within unlabeled data containing unknowns, improving classification accuracy and reducing annotation costs.

Contribution

The paper presents the first active learning method specifically designed for open-set annotation, utilizing a Gaussian Mixture Model and temperature scaling to enhance known class detection.

Findings

Significantly improves selection quality of known class examples.

Achieves higher classification accuracy with lower annotation costs.

Outperforms state-of-the-art active learning methods in open-set scenarios.

Abstract

Existing active learning studies typically work in the closed-set setting by assuming that all data examples to be labeled are drawn from known classes. However, in real annotation tasks, the unlabeled data usually contains a large amount of examples from unknown classes, resulting in the failure of most active learning methods. To tackle this open-set annotation (OSA) problem, we propose a new active learning framework called LfOSA, which boosts the classification performance with an effective sampling strategy to precisely detect examples from known classes for annotation. The LfOSA framework introduces an auxiliary network to model the per-example max activation value (MAV) distribution with a Gaussian Mixture Model, which can dynamically select the examples with highest probability from known classes in the unlabeled set. Moreover, by reducing the temperature $T$ of the loss function, the detection model will be further optimized by exploiting both known and unknown supervision. The experimental results show that the proposed method can significantly improve the selection quality of known classes, and achieve higher classification accuracy with lower annotation cost than state-of-the-art active learning methods. To the best of our knowledge, this is the first work of active learning for open-set annotation.