SAMCLR: Contrastive pre-training on complex scenes using SAM for view sampling

TL;DR

SAMCLR enhances contrastive pre-training on complex scenes by integrating SAM for semantic segmentation, enabling more effective view sampling from the same regions, leading to improved downstream classification performance.

Contribution

Introduces SAMCLR, a novel extension of SimCLR that uses SAM for semantic segmentation to improve contrastive learning on complex scenes.

Findings

SAMCLR outperforms SimCLR, DINO, and MoCo on several benchmarks.

Pre-training on Cityscapes and ADE20K improves classification accuracy.

Semantic region sampling enhances contrastive learning effectiveness.

Abstract

In Computer Vision, self-supervised contrastive learning enforces similar representations between different views of the same image. The pre-training is most often performed on image classification datasets, like ImageNet, where images mainly contain a single class of objects. However, when dealing with complex scenes with multiple items, it becomes very unlikely for several views of the same image to represent the same object category. In this setting, we propose SAMCLR, an add-on to SimCLR which uses SAM to segment the image into semantic regions, then sample the two views from the same region. Preliminary results show empirically that when pre-training on Cityscapes and ADE20K, then evaluating on classification on CIFAR-10, STL10 and ImageNette, SAMCLR performs at least on par with, and most often significantly outperforms not only SimCLR, but also DINO and MoCo.