Big Self-Supervised Models are Strong Semi-Supervised Learners

TL;DR

This paper demonstrates that large self-supervised models, when fine-tuned with limited labeled data, can serve as highly effective semi-supervised learners, significantly reducing label requirements for ImageNet classification.

Contribution

It introduces a semi-supervised learning approach combining big models, self-supervised pretraining, and distillation, achieving state-of-the-art label efficiency on ImageNet.

Findings

Achieves 73.9% top-1 accuracy with only 1% labels on ImageNet.

Outperforms supervised training with all labels at 10% labels.

Big models benefit more from unlabeled data when fewer labels are available.

Abstract

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels ($\le$13 labeled images per class) using ResNet-50, a $10\times$ improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.