Active Learning of Deep Neural Networks via Gradient-Free Cutting Planes

TL;DR

This paper introduces a novel gradient-free cutting-plane method for training deep neural networks in an active learning context, providing convergence guarantees and demonstrating improved sample efficiency over existing methods.

Contribution

It extends cutting-plane algorithms to deep neural networks and develops the first deep active learning scheme with proven convergence guarantees.

Findings

Demonstrates geometric contraction rate of the feasible set.

Achieves better sample efficiency than baseline methods.

Validates effectiveness on synthetic and real datasets.

Abstract

Active learning methods aim to improve sample complexity in machine learning. In this work, we investigate an active learning scheme via a novel gradient-free cutting-plane training method for ReLU networks of arbitrary depth and develop a convergence theory. We demonstrate, for the first time, that cutting-plane algorithms, traditionally used in linear models, can be extended to deep neural networks despite their nonconvexity and nonlinear decision boundaries. Moreover, this training method induces the first deep active learning scheme known to achieve convergence guarantees, revealing a geometric contraction rate of the feasible set. We exemplify the effectiveness of our proposed active learning method against popular deep active learning baselines via both synthetic data experiments and sentimental classification task on real datasets.