Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning

TL;DR

Meta-Query-Net addresses the challenge of balancing purity and informativeness in open-set active learning by adaptively optimizing sample selection, leading to significant accuracy improvements over existing methods.

Contribution

The paper introduces MQ-Net, a novel approach that adaptively balances purity and informativeness without requiring extra validation data, using skyline regularization.

Findings

Achieves 20.14% accuracy improvement over state-of-the-art methods.

Effectively captures dominance relationships among unlabeled examples.

Demonstrates robustness across multiple open-set active learning scenarios.

Abstract

Unlabeled data examples awaiting annotations contain open-set noise inevitably. A few active learning studies have attempted to deal with this open-set noise for sample selection by filtering out the noisy examples. However, because focusing on the purity of examples in a query set leads to overlooking the informativeness of the examples, the best balancing of purity and informativeness remains an important question. In this paper, to solve this purity-informativeness dilemma in open-set active learning, we propose a novel Meta-Query-Net,(MQ-Net) that adaptively finds the best balancing between the two factors. Specifically, by leveraging the multi-round property of active learning, we train MQ-Net using a query set without an additional validation set. Furthermore, a clear dominance relationship between unlabeled examples is effectively captured by MQ-Net through a novel skyline regularization. Extensive experiments on multiple open-set active learning scenarios demonstrate that the proposed MQ-Net achieves 20.14% improvement in terms of accuracy, compared with the state-of-the-art methods.