Enhancing Contrastive Learning with Efficient Combinatorial Positive Pairing

TL;DR

This paper introduces ECPP, a multi-view strategy that enhances contrastive learning efficiency and performance by optimizing view combinations and sampling, achieving state-of-the-art results on CIFAR-10 and ImageNet-100.

Contribution

The paper proposes ECPP, a novel multi-view approach that improves contrastive learning speed and accuracy, applicable to various methods like SimCLR.

Findings

ECPP increases learning speed by $_{K} ext{C}_2$ times for small learning rates.

ECPP achieves state-of-the-art performance on CIFAR-10 and ImageNet-100.

ECPP outperforms supervised learning on ImageNet-100.

Abstract

In the past few years, contrastive learning has played a central role for the success of visual unsupervised representation learning. Around the same time, high-performance non-contrastive learning methods have been developed as well. While most of the works utilize only two views, we carefully review the existing multi-view methods and propose a general multi-view strategy that can improve learning speed and performance of any contrastive or non-contrastive method. We first analyze CMC's full-graph paradigm and empirically show that the learning speed of $K$-views can be increased by $_{K}\mathrm{C}_{2}$ times for small learning rate and early training. Then, we upgrade CMC's full-graph by mixing views created by a crop-only augmentation, adopting small-size views as in SwAV multi-crop, and modifying the negative sampling. The resulting multi-view strategy is called ECPP (Efficient Combinatorial Positive Pairing). We investigate the effectiveness of ECPP by applying it to SimCLR and assessing the linear evaluation performance for CIFAR-10 and ImageNet-100. For each benchmark, we achieve a state-of-the-art performance. In case of ImageNet-100, ECPP boosted SimCLR outperforms supervised learning.