SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learning

TL;DR

This paper introduces SST, a semi-supervised learning framework with a self-adaptive thresholding mechanism that dynamically selects high-quality pseudo-labels, leading to state-of-the-art results with high efficiency and scalability.

Contribution

The paper proposes a novel Self-Adaptive Thresholding (SAT) mechanism within SST that adjusts thresholds based on learning progress, improving pseudo-label quality and training efficiency.

Findings

SST achieves top performance on ImageNet-1K SSL benchmarks.

Semi-SST-ViT-Huge reaches 80.7% / 84.9% accuracy with only 1% / 10% labeled data.

SST outperforms fully-supervised models using significantly less labeled data.

Abstract

Neural networks have demonstrated exceptional performance in supervised learning, benefiting from abundant high-quality annotated data. However, obtaining such data in real-world scenarios is costly and labor-intensive. Semi-supervised learning (SSL) offers a solution to this problem. Recent studies, such as Semi-ViT and Noisy Student, which employ consistency regularization or pseudo-labeling, have demonstrated significant achievements. However, they still face challenges, particularly in accurately selecting sufficient high-quality pseudo-labels due to their reliance on fixed thresholds. Recent methods such as FlexMatch and FreeMatch have introduced flexible or self-adaptive thresholding techniques, greatly advancing SSL research. Nonetheless, their process of updating thresholds at each iteration is deemed time-consuming, computationally intensive, and potentially unnecessary. To address these issues, we propose Self-training with Self-adaptive Thresholding (SST), a novel, effective, and efficient SSL framework. SST introduces an innovative Self-Adaptive Thresholding (SAT) mechanism that adaptively adjusts class-specific thresholds based on the model's learning progress. SAT ensures the selection of high-quality pseudo-labeled data, mitigating the risks of inaccurate pseudo-labels and confirmation bias. Extensive experiments demonstrate that SST achieves state-of-the-art performance with remarkable efficiency, generalization, and scalability across various architectures and datasets. Semi-SST-ViT-Huge achieves the best results on competitive ImageNet-1K SSL benchmarks, with 80.7% / 84.9% Top-1 accuracy using only 1% / 10% labeled data. Compared to the fully-supervised DeiT-III-ViT-Huge, which achieves 84.8% Top-1 accuracy using 100% labeled data, our method demonstrates superior performance using only 10% labeled data.