On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?

TL;DR

This paper investigates the interplay between active learning and self-supervised semi-supervised learning, finding that self-supervised pretraining enhances semi-supervised methods but diminishes the additional benefits of active learning.

Contribution

It introduces a novel framework combining self-supervised pretraining, active learning, and self-training, and systematically evaluates their individual and combined effects.

Findings

Self-supervised pretraining improves semi-supervised learning in low-label regimes.

Active learning benefits are reduced when combined with self-supervised techniques.

State-of-the-art active learning algorithms offer no additional gains with advanced S4L methods.

Abstract

Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g.~Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.