Improved Baselines with Momentum Contrastive Learning

TL;DR

This paper enhances the MoCo framework for contrastive unsupervised learning by incorporating design improvements from SimCLR, such as an MLP projection head and increased data augmentation, leading to stronger, more accessible baselines.

Contribution

It demonstrates that simple modifications to MoCo can outperform SimCLR without large batch sizes, improving baseline performance in contrastive learning.

Findings

Stronger MoCo baselines surpass SimCLR performance.

No need for large training batches with modifications.

Enhanced unsupervised learning accessibility.

Abstract

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.