Saliency Guided Contrastive Learning on Scene Images

TL;DR

This paper introduces a saliency-guided contrastive learning method that enhances self-supervised learning on complex scene images by focusing on discriminative regions, leading to improved performance in various evaluation settings.

Contribution

It proposes using saliency maps to identify and emphasize important regions in scene images during contrastive learning, a novel approach for better representation learning from less-curated data.

Findings

Achieved +1.1% Top1 accuracy in ImageNet linear evaluation.

Improved semi-supervised learning accuracy by +4.3% with 1% labels.

Enhanced contrastive learning performance on scene images.

Abstract

Self-supervised learning holds promise in leveraging large numbers of unlabeled data. However, its success heavily relies on the highly-curated dataset, e.g., ImageNet, which still needs human cleaning. Directly learning representations from less-curated scene images is essential for pushing self-supervised learning to a higher level. Different from curated images which include simple and clear semantic information, scene images are more complex and mosaic because they often include complex scenes and multiple objects. Despite being feasible, recent works largely overlooked discovering the most discriminative regions for contrastive learning to object representations in scene images. In this work, we leverage the saliency map derived from the model's output during learning to highlight these discriminative regions and guide the whole contrastive learning. Specifically, the saliency map first guides the method to crop its discriminative regions as positive pairs and then reweighs the contrastive losses among different crops by its saliency scores. Our method significantly improves the performance of self-supervised learning on scene images by +1.1, +4.3, +2.2 Top1 accuracy in ImageNet linear evaluation, Semi-supervised learning with 1% and 10% ImageNet labels, respectively. We hope our insights on saliency maps can motivate future research on more general-purpose unsupervised representation learning from scene data.