Bag of Instances Aggregation Boosts Self-supervised Distillation

TL;DR

The paper introduces BINGO, a distillation method that leverages relationships among similar samples to improve self-supervised learning, especially for small-scale models, achieving state-of-the-art results on ImageNet.

Contribution

Proposes BINGO, a novel distillation strategy that transfers sample relationships from teacher to student to enhance self-supervised learning performance.

Findings

BINGO outperforms baselines on small-scale models.

Achieves 65.5% and 68.9% top-1 accuracy on ImageNet with ResNet-18 and ResNet-34.

Significant improvement over existing methods.

Abstract

Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well as its augmentations as an individual class and try to distinguish them from all other images. However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization. This is especially true for small-scale models, in which we find the performance drops dramatically comparing with its supervised counterpart. In this paper, we propose a simple but effective distillation strategy for unsupervised learning. The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance. Our method, termed as BINGO, which is short for Bag of InstaNces aGgregatiOn, targets at transferring the relationship learned by the teacher to the student. Here bag of instances indicates a set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation is to aggregate compact representations over the student with respect to instances in a bag. Notably, BINGO achieves new state-of-the-art performance on small-scale models, i.e., 65.5% and 68.9% top-1 accuracies with linear evaluation on ImageNet, using ResNet-18 and ResNet-34 as the backbones respectively, surpassing baselines (52.5% and 57.4% top-1 accuracies) by a significant margin. The code is available at https://github.com/haohang96/bingo.