Nuclear Discrepancy for Active Learning

TL;DR

This paper introduces the Nuclear Discrepancy bound for active learning, demonstrating that looser, probabilistically motivated bounds can outperform tighter bounds in practical scenarios.

Contribution

It proposes a new Nuclear Discrepancy bound for active learning, showing its effectiveness over existing bounds through theoretical analysis and empirical validation.

Findings

Nuclear Discrepancy bound outperforms tighter bounds in practice

Looser bounds focusing on realistic scenarios can lead to better active learning

Empirical results confirm the probabilistic motivation behind the new bound

Abstract

Active learning algorithms propose which unlabeled objects should be queried for their labels to improve a predictive model the most. We study active learners that minimize generalization bounds and uncover relationships between these bounds that lead to an improved approach to active learning. In particular we show the relation between the bound of the state-of-the-art Maximum Mean Discrepancy (MMD) active learner, the bound of the Discrepancy, and a new and looser bound that we refer to as the Nuclear Discrepancy bound. We motivate this bound by a probabilistic argument: we show it considers situations which are more likely to occur. Our experiments indicate that active learning using the tightest Discrepancy bound performs the worst in terms of the squared loss. Overall, our proposed loosest Nuclear Discrepancy generalization bound performs the best. We confirm our probabilistic argument empirically: the other bounds focus on more pessimistic scenarios that are rarer in practice. We conclude that tightness of bounds is not always of main importance and that active learning methods should concentrate on realistic scenarios in order to improve performance.