Deep Individual Active Learning: Safeguarding against Out-of-Distribution Challenges in Neural Networks

TL;DR

This paper introduces a new active learning method that improves model training efficiency, especially when test data differs from training data.

Contribution

The novel contribution is an efficient algorithm for minimizing min-max regret in active learning, reducing training data needs.

Findings

The proposed algorithm reduces required training set size by up to 15.4% for CIFAR10.

It achieves 35.1% reduction in training data for MNIST with out-of-distribution challenges.

The method is computationally efficient for neural networks in individual active learning settings.

Abstract

Active learning (AL) is a paradigm focused on purposefully selecting training data to enhance a model’s performance by minimizing the need for annotated samples. Typically, strategies assume that the training pool shares the same distribution as the test set, which is not always valid in privacy-sensitive applications where annotating user data is challenging. In this study, we operate within an individual setting and leverage an active learning criterion which selects data points for labeling based on minimizing the min-max regret on a small unlabeled test set sample. Our key contribution lies in the development of an efficient algorithm, addressing the challenging computational complexity associated with approximating this criterion for neural networks. Notably, our results show that, especially in the presence of out-of-distribution data, the proposed algorithm substantially reduces the required training set size by up to 15.4%, 11%, and 35.1% for CIFAR10, EMNIST, and MNIST datasets, respectively.

Figures