A Neural Pre-Conditioning Active Learning Algorithm to Reduce Label Complexity

TL;DR

This paper introduces a neural pre-conditioning active learning algorithm that enhances label efficiency by selecting diverse, informative samples, and proposes a new evaluation method based on SSL performance improvements.

Contribution

The paper presents a novel neural pre-conditioning (NPC) active learning algorithm inspired by NTK analysis, improving label efficiency and proposing a new evaluation framework based on SSL performance.

Findings

NPC improves downstream training in the large-width regime.

Combining NPC with SSL achieves high performance with few labels.

The proposed evaluation method better reflects label efficiency.

Abstract

Deep learning (DL) algorithms rely on massive amounts of labeled data. Semi-supervised learning (SSL) and active learning (AL) aim to reduce this label complexity by leveraging unlabeled data or carefully acquiring labels, respectively. In this work, we primarily focus on designing an AL algorithm but first argue for a change in how AL algorithms should be evaluated. Although unlabeled data is readily available in pool-based AL, AL algorithms are usually evaluated by measuring the increase in supervised learning (SL) performance at consecutive acquisition steps. Because this measures performance gains from both newly acquired instances and newly acquired labels, we propose to instead evaluate the label efficiency of AL algorithms by measuring the increase in SSL performance at consecutive acquisition steps. After surveying tools that can be used to this end, we propose our neural pre-conditioning (NPC) algorithm inspired by a Neural Tangent Kernel (NTK) analysis. Our algorithm incorporates the classifier's uncertainty on unlabeled data and penalizes redundant samples within candidate batches to efficiently acquire a diverse set of informative labels. Furthermore, we prove that NPC improves downstream training in the large-width regime in a manner previously observed to correlate with generalization. Comparisons with other AL algorithms show that a state-of-the-art SSL algorithm coupled with NPC can achieve high performance using very few labeled data.