Deep Active Learning over the Long Tail

TL;DR

This paper introduces a novel deep active learning algorithm that leverages farthest-first traversal in neural activation space, significantly reducing sample complexity and outperforming traditional uncertainty sampling on multiple datasets.

Contribution

The paper presents a new active learning method based on farthest-first traversal in neural activation space, improving sample efficiency over passive and uncertainty sampling methods.

Findings

Significant reduction in sample complexity on MNIST, CIFAR-10, and CIFAR-100.

Outperforms traditional uncertainty sampling techniques.

Identifies limitations of uncertainty sampling in certain cases.

Abstract

This paper is concerned with pool-based active learning for deep neural networks. Motivated by coreset dataset compression ideas, we present a novel active learning algorithm that queries consecutive points from the pool using farthest-first traversals in the space of neural activation over a representation layer. We show consistent and overwhelming improvement in sample complexity over passive learning (random sampling) for three datasets: MNIST, CIFAR-10, and CIFAR-100. In addition, our algorithm outperforms the traditional uncertainty sampling technique (obtained using softmax activations), and we identify cases where uncertainty sampling is only slightly better than random sampling.