Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime

TL;DR

This paper demonstrates that target pretraining enhances semi-supervised learning performance in low-label scenarios by addressing covariate shift, with extensive experiments validating its effectiveness across classification and segmentation tasks.

Contribution

It introduces a contrastive target pretraining step before semi-supervised finetuning, improving SSL in low-label regimes by mitigating covariate shift and leveraging pretrained weights.

Findings

Target pretraining improves SSL performance in low-label regimes.

Pretrained weights significantly contribute to state-of-the-art results.

Extensive experiments confirm the effectiveness across tasks.

Abstract

Semi-supervised learning (SSL) addresses the lack of labeled data by exploiting large unlabeled data through pseudolabeling. However, in the extremely low-label regime, pseudo labels could be incorrect, a.k.a. the confirmation bias, and the pseudo labels will in turn harm the network training. Recent studies combined finetuning (FT) from pretrained weights with SSL to mitigate the challenges and claimed superior results in the low-label regime. In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off-the-shelf semi-supervised learners. We further argue that direct finetuning from pretrained weights is suboptimal due to covariate shift and propose a contrastive target pretraining step to adapt model weights towards target dataset. We carried out extensive experiments on both classification and segmentation tasks by doing target pretraining then followed by semi-supervised finetuning. The promising results validate the efficacy of target pretraining for SSL, in particular in the low-label regime.