TL;DR
This paper compares semi-supervised learning and pre-trained models in image classification, finding that pre-trained models outperform SSL in large-model regimes, and discusses future directions for SSL research.
Contribution
It provides a fair comparison between SSL and pre-trained models, revealing the dominance of pre-trained models and highlighting the need for new SSL approaches.
Findings
Pre-trained models outperform SSL on standard benchmarks.
SSL has reached its 'Waterloo' in the era of large models.
Multi-Modal Large Language Models still face significant performance challenges.
Abstract
Semi-supervised learning (SSL) alleviates the cost of data labeling process by exploiting unlabeled data and has achieved promising results. Meanwhile, with the development of large foundation models, exploiting pre-trained models becomes a promising way to address the label scarcity in the downstream tasks, such as various parameter-efficient fine-tuning techniques. This raises a natural yet critical question: When labeled data is limited, should we rely on unlabeled data or pre-trained models? To investigate this issue, we conduct a fair comparison between SSL methods and pre-trained models (e.g., CLIP) on representative image classification tasks under a controlled supervision budget. Experiments reveal that SSL has met its ``Waterloo" in the era of large models, as pre-trained models show both high efficiency and strong performance on widely adopted SSL benchmarks. This underscores…
Peer Reviews
Decision·Submitted to ICLR 2026
The paper presents comprehensive experiments.
1. This paper focuses solely on experimental comparisons without proposing any new method. Therefore, the experimental setup is critical and should be described in detail in the main text. 2. The comparison settings are entirely unfair. For example, traditional SSL methods use ResNet backbones (e.g., ResNet-28-2), whereas prompt-tuning methods use ViT/B-16. Moreover, comparing SSL and few-shot learning is conceptually unsound. SSL with CLIP has already been explored, and under such unfair exper
This is a timely study in the era of large models and has the potential to deliver useful insight for the SSL community. The experiments are relatively comprehensive, covering main settings in SSL research. Takeaways are clearly stated.
The benchmark datasets—CIFAR-10/100, STL-10, and ImageNet—may overlap semantically with VLM pretraining data, potentially advantaging the VLMs in this comparison. By contrast, in domains where the pre-trained model has seen less data (e.g., medical imaging), it is unclear whether the conclusions would still hold. Experiments probing this low-overlap setting are currently missing.
- The problem studied, i.e., comparing unlabeled data and pre-trained models, is interesting and important to the literature. - The experiments are comprehensive and cover various semi-supervised learning settings, including standard, open-set, open-world, and long-tailed semi-supervised learning. Therefore, the conclusions are convincing. - The writing is excellent, making the paper easy to understand. - This paper is an important position paper for the field of weakly supervised learning.
- There is a lack of related work discussing the use of zero-shot CLIP for traditional weakly supervised learning, a more general form of semi-supervised learning. Additionally, some work discusses semi-supervised learning with CLIP. - The title would be better with "semi-supervised learning" instead of "SSL" since "SSL" has multiple meanings, such as self-supervised learning. - In experiments on standard semi-supervised learning, SOTA SSL algorithms such as FixMatch seem to achieve better per
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsSoftmax · Attention Is All You Need
