Distilling Visual Priors from Self-Supervised Learning

TL;DR

This paper introduces a two-phase approach combining self-supervised learning and knowledge distillation to enhance CNN generalization on small datasets, featuring a novel margin loss for contrastive learning.

Contribution

It proposes a new pipeline that distills visual priors from self-supervised models into CNNs, improving performance in data-limited image classification tasks.

Findings

Achieves competitive results in VIPriors challenge.

Introduces a novel margin loss for contrastive learning.

Demonstrates improved generalization on small datasets.

Abstract

Convolutional Neural Networks (CNNs) are prone to overfit small training datasets. We present a novel two-phase pipeline that leverages self-supervised learning and knowledge distillation to improve the generalization ability of CNN models for image classification under the data-deficient setting. The first phase is to learn a teacher model which possesses rich and generalizable visual representations via self-supervised learning, and the second phase is to distill the representations into a student model in a self-distillation manner, and meanwhile fine-tune the student model for the image classification task. We also propose a novel margin loss for the self-supervised contrastive learning proxy task to better learn the representation under the data-deficient scenario. Together with other tricks, we achieve competitive performance in the VIPriors image classification challenge.