Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR

TL;DR

This paper analyzes SimCLR, a leading semi-supervised learning framework, focusing on contrast learning properties, knowledge distillation, and transfer learning, revealing insights into their effects on model performance and data set characteristics.

Contribution

The paper provides a detailed analysis of SimCLR's contrast learning, explores knowledge distillation with teacher-student models, and examines transfer learning's effectiveness relative to class count.

Findings

Contrast learning significantly impacts fine-tuning performance.

Knowledge distillation benefits from shared base models.

Transfer learning performs better with fewer classes.

Abstract

Recent breakthroughs in the field of semi-supervised learning have achieved results that match state-of-the-art traditional supervised learning methods. Most successful semi-supervised learning approaches in computer vision focus on leveraging huge amount of unlabeled data, learning the general representation via data augmentation and transformation, creating pseudo labels, implementing different loss functions, and eventually transferring this knowledge to more task-specific smaller models. In this paper, we aim to conduct our analyses on three different aspects of SimCLR, the current state-of-the-art semi-supervised learning framework for computer vision. First, we analyze properties of contrast learning on fine-tuning, as we understand that contrast learning is what makes this method so successful. Second, we research knowledge distillation through teacher-forcing paradigm. We observe that when the teacher and the student share the same base model, knowledge distillation will achieve better result. Finally, we study how transfer learning works and its relationship with the number of classes on different data sets. Our results indicate that transfer learning performs better when number of classes are smaller.