Towards Democratizing Joint-Embedding Self-Supervised Learning

TL;DR

This paper challenges common misconceptions in joint-embedding self-supervised learning, demonstrating that simpler, less resource-intensive methods can achieve competitive results, and provides an optimized PyTorch library to facilitate broader research and evaluation.

Contribution

It debunks prevalent myths in JE-SSL, showing that effective representations can be learned with minimal data augmentation and resources, and introduces a user-friendly PyTorch library for easier experimentation.

Findings

Simpler data augmentation suffices for effective JE-SSL.

Training with a single negative example can still learn useful representations.

Misconceptions about batch size and augmentation requirements are largely unfounded.

Abstract

Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data. The development of JE-SSL methods was driven primarily by the search for ever increasing downstream classification accuracies, using huge computational resources, and typically built upon insights and intuitions inherited from a close parent JE-SSL method. This has led unwittingly to numerous pre-conceived ideas that carried over across methods e.g. that SimCLR requires very large mini batches to yield competitive accuracies; that strong and computationally slow data augmentations are required. In this work, we debunk several such ill-formed a priori ideas in the hope to unleash the full potential of JE-SSL free of unnecessary limitations. In fact, when carefully evaluating performances across different downstream tasks and properly optimizing hyper-parameters of the methods, we most often -- if not always -- see that these widespread misconceptions do not hold. For example we show that it is possible to train SimCLR to learn useful representations, while using a single image patch as negative example, and simple Gaussian noise as the only data augmentation for the positive pair. Along these lines, in the hope to democratize JE-SSL and to allow researchers to easily make more extensive evaluations of their methods, we introduce an optimized PyTorch library for SSL.