Iterative Pseudo-Labeling with Deep Feature Annotation and Confidence-Based Sampling

TL;DR

This paper enhances pseudo-labeling for deep neural networks by introducing a confidence-based sampling method that reduces annotation effort and improves performance without needing a validation set.

Contribution

It proposes a confidence-based sampling strategy for iterative pseudo-labeling that minimizes annotation effort and eliminates the need for a validation set, improving over existing methods.

Findings

Confidence DeepFA outperforms baseline and original DeepFA.

Reduces annotation effort to dozens of samples per class.

Effective without a validation set.

Abstract

Training deep neural networks is challenging when large and annotated datasets are unavailable. Extensive manual annotation of data samples is time-consuming, expensive, and error-prone, notably when it needs to be done by experts. To address this issue, increased attention has been devoted to techniques that propagate uncertain labels (also called pseudo labels) to large amounts of unsupervised samples and use them for training the model. However, these techniques still need hundreds of supervised samples per class in the training set and a validation set with extra supervised samples to tune the model. We improve a recent iterative pseudo-labeling technique, Deep Feature Annotation (DeepFA), by selecting the most confident unsupervised samples to iteratively train a deep neural network. Our confidence-based sampling strategy relies on only dozens of annotated training samples per class with no validation set, considerably reducing user effort in data annotation. We first ascertain the best configuration for the baseline -- a self-trained deep neural network -- and then evaluate our confidence DeepFA for different confidence thresholds. Experiments on six datasets show that DeepFA already outperforms the self-trained baseline, but confidence DeepFA can considerably outperform the original DeepFA and the baseline.