A Studious Approach to Semi-Supervised Learning

TL;DR

This paper investigates distillation in semi-supervised learning, demonstrating that smaller student models trained with soft labels from a teacher can outperform baseline models, especially with fewer labels, enhancing deployability.

Contribution

It presents an ablation study showing how distillation reduces model size and improves performance in semi-supervised learning for computer vision tasks.

Findings

Smaller student networks benefit more from distillation with fewer labels.

Distillation improves generalization over baseline supervised models.

Reduced-parameter models maintain or improve accuracy.

Abstract

The problem of learning from few labeled examples while using large amounts of unlabeled data has been approached by various semi-supervised methods. Although these methods can achieve superior performance, the models are often not deployable due to the large number of parameters. This paper is an ablation study of distillation in a semi-supervised setting, which not just reduces the number of parameters of the model but can achieve this while improving the performance over the baseline supervised model and making it better at generalizing. After the supervised pretraining, the network is used as a teacher model, and a student network is trained over the soft labels that the teacher model generates over the entire unlabeled data. We find that the fewer the labels, the more this approach benefits from a smaller student network. This brings forward the potential of distillation as an effective solution to enhance performance in semi-supervised computer vision tasks while maintaining deployability.