MixCo: Mix-up Contrastive Learning for Visual Representation

TL;DR

MixCo introduces a novel semi-positives contrastive learning method using mix-up of images, improving visual representation quality especially for smaller models in self-supervised learning tasks.

Contribution

The paper proposes MixCo, a new contrastive learning approach that incorporates mix-up of images to enhance semi-supervised visual representations.

Findings

MixCo improves test accuracy across datasets.

Greater benefits observed for limited-capacity models.

Consistent performance gains in self-supervised learning.

Abstract

Contrastive learning has shown remarkable results in recent self-supervised approaches for visual representation. By learning to contrast positive pairs' representation from the corresponding negatives pairs, one can train good visual representations without human annotations. This paper proposes Mix-up Contrast (MixCo), which extends the contrastive learning concept to semi-positives encoded from the mix-up of positive and negative images. MixCo aims to learn the relative similarity of representations, reflecting how much the mixed images have the original positives. We validate the efficacy of MixCo when applied to the recent self-supervised learning algorithms under the standard linear evaluation protocol on TinyImageNet, CIFAR10, and CIFAR100. In the experiments, MixCo consistently improves test accuracy. Remarkably, the improvement is more significant when the learning capacity (e.g., model size) is limited, suggesting that MixCo might be more useful in real-world scenarios. The code is available at: https://github.com/Lee-Gihun/MixCo-Mixup-Contrast.