Resource and data efficient self supervised learning

TL;DR

This paper explores a two-step pretraining approach for contrastive self-supervised learning, improving efficiency and performance on small or less diverse datasets across natural and medical images.

Contribution

It introduces weight scaling and filter selection methods to enhance pretraining, enabling effective fine-tuning with smaller datasets, batch sizes, and training times.

Findings

Double pretraining improves downstream performance.

Faster convergence and smaller resource requirements.

Effective across multiple contrastive learning methods.

Abstract

We investigate the utility of pretraining by contrastive self supervised learning on both natural-scene and medical imaging datasets when the unlabeled dataset size is small, or when the diversity within the unlabeled set does not lead to better representations. We use a two step approach which is analogous to supervised training with ImageNet initialization, where we pretrain networks that are already pretrained on ImageNet dataset to improve downstream task performance on the domain of interest. To improve the speed of convergence and the overall performance, we propose weight scaling and filter selection methods prior to second step of pretraining. We demonstrate the utility of this approach on three popular contrastive techniques, namely SimCLR, SWaV and BYOL. Benefits of double pretraining include better performance, faster convergence, ability to train with smaller batch sizes and smaller image dimensions with negligible differences in performance. We hope our work helps democratize self-supervision by enabling researchers to fine-tune models without requiring large clusters or long training times.