Contrasting Contrastive Self-Supervised Representation Learning Pipelines

TL;DR

This paper provides a comprehensive analysis of contrastive self-supervised learning methods, examining how training algorithms, datasets, and tasks influence performance, supported by extensive experiments and a new benchmark.

Contribution

It offers a detailed empirical study of contrastive self-supervised learning, including a large-scale experiment analysis and the introduction of the Visual Representation Benchmark (ViRB).

Findings

Contrastive models often outperform supervised models on downstream tasks.

Pre-training datasets significantly impact end task performance.

Current benchmarks may not fully capture model capabilities.

Abstract

In the past few years, we have witnessed remarkable breakthroughs in self-supervised representation learning. Despite the success and adoption of representations learned through this paradigm, much is yet to be understood about how different training methods and datasets influence performance on downstream tasks. In this paper, we analyze contrastive approaches as one of the most successful and popular variants of self-supervised representation learning. We perform this analysis from the perspective of the training algorithms, pre-training datasets and end tasks. We examine over 700 training experiments including 30 encoders, 4 pre-training datasets and 20 diverse downstream tasks. Our experiments address various questions regarding the performance of self-supervised models compared to their supervised counterparts, current benchmarks used for evaluation, and the effect of the pre-training data on end task performance. Our Visual Representation Benchmark (ViRB) is available at: https://github.com/allenai/virb.