Uncertainty Herding: One Active Learning Method for All Label Budgets

TL;DR

This paper introduces Uncertainty Herding, a versatile active learning method that adapts seamlessly across different label budget regimes, outperforming existing methods in both low- and high-budget scenarios.

Contribution

It proposes uncertainty coverage as a unified objective and develops Uncertainty Herding, a simple, fast, and nearly optimal method that works well across all label budgets.

Findings

Outperforms state-of-the-art in various tasks

Works effectively in both low- and high-budget regimes

Nearly optimizes distribution-level coverage

Abstract

Most active learning research has focused on methods which perform well when many labels are available, but can be dramatically worse than random selection when label budgets are small. Other methods have focused on the low-budget regime, but do poorly as label budgets increase. As the line between "low" and "high" budgets varies by problem, this is a serious issue in practice. We propose uncertainty coverage, an objective which generalizes a variety of low- and high-budget objectives, as well as natural, hyperparameter-light methods to smoothly interpolate between low- and high-budget regimes. We call greedy optimization of the estimate Uncertainty Herding; this simple method is computationally fast, and we prove that it nearly optimizes the distribution-level coverage. In experimental validation across a variety of active learning tasks, our proposal matches or beats state-of-the-art performance in essentially all cases; it is the only method of which we are aware that reliably works well in both low- and high-budget settings.