A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

TL;DR

This paper introduces a comprehensive framework for contrastive self-supervised learning (CSL), analyzes existing methods within it, and proposes a new approach, YADIM, demonstrating competitive results and robustness.

Contribution

The paper presents a unifying conceptual framework for CSL, analyzes leading approaches under this framework, and designs a new method, YADIM, with improved robustness and performance.

Findings

YADIM achieves competitive results on CIFAR-10, STL-10, and ImageNet.

The framework unifies diverse CSL approaches under five key aspects.

YADIM is more robust to encoder and representation choices.

Abstract

Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset. We present a conceptual framework that characterizes CSL approaches in five aspects (1) data augmentation pipeline, (2) encoder selection, (3) representation extraction, (4) similarity measure, and (5) loss function. We analyze three leading CSL approaches--AMDIM, CPC, and SimCLR--, and show that despite different motivations, they are special cases under this framework. We show the utility of our framework by designing Yet Another DIM (YADIM) which achieves competitive results on CIFAR-10, STL-10 and ImageNet, and is more robust to the choice of encoder and the representation extraction strategy. To support ongoing CSL research, we release the PyTorch implementation of this conceptual framework along with standardized implementations of AMDIM, CPC (V2), SimCLR, BYOL, Moco (V2) and YADIM.