On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning

TL;DR

This paper introduces a more efficient self-supervised learning method that reduces computational costs and improves performance, especially for small models and limited data, by using a novel self-distillation loss and corrected memory bank updates.

Contribution

It proposes a single-branch SSL approach with a new self-distillation loss and improved memory bank update rule, enhancing efficiency and convergence speed.

Findings

Outperforms baselines with less training overhead

Effective for small models and limited data

Accelerates convergence significantly

Abstract

Self-supervised learning (SSL) has developed rapidly in recent years. However, most of the mainstream methods are computationally expensive and rely on two (or more) augmentations for each image to construct positive pairs. Moreover, they mainly focus on large models and large-scale datasets, which lack flexibility and feasibility in many practical applications. In this paper, we propose an efficient single-branch SSL method based on non-parametric instance discrimination, aiming to improve the algorithm, model, and data efficiency of SSL. By analyzing the gradient formula, we correct the update rule of the memory bank with improved performance. We further propose a novel self-distillation loss that minimizes the KL divergence between the probability distribution and its square root version. We show that this alleviates the infrequent updating problem in instance discrimination and greatly accelerates convergence. We systematically compare the training overhead and performance of different methods in different scales of data, and under different backbones. Experimental results show that our method outperforms various baselines with significantly less overhead, and is especially effective for limited amounts of data and small models.