Active Learning with Neural Networks: Insights from Nonparametric Statistics

TL;DR

This paper provides the first rigorous theoretical guarantees for deep active learning with neural networks, showing near-optimal label complexity under standard conditions and polylogarithmic complexity with abstention.

Contribution

It introduces the first near-optimal label complexity guarantees for deep active learning, connecting neural networks with nonparametric statistics and extending results to Radon BV spaces.

Findings

Achieves minimax label complexity under low noise conditions.

Develops an efficient active learning algorithm with polylogarithmic label complexity with abstention.

Extends theoretical guarantees to Radon BV spaces.

Abstract

Deep neural networks have great representation power, but typically require large numbers of training examples. This motivates deep active learning methods that can significantly reduce the amount of labeled training data. Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive. This constitutes a significant gap between theory and practice. This paper tackles this gap by providing the first near-optimal label complexity guarantees for deep active learning. The key insight is to study deep active learning from the nonparametric classification perspective. Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefficient and other logarithmic terms. When equipped with an abstention option, we further develop an efficient deep active learning algorithm that achieves $\mathsf{polylog}(\frac{1}{\epsilon})$ label complexity, without any low noise assumptions. We also provide extensions of our results beyond the commonly studied Sobolev/H\"older spaces and develop label complexity guarantees for learning in Radon $\mathsf{BV}^2$ spaces, which have recently been proposed as natural function spaces associated with neural networks.