Exploring Localization for Self-supervised Fine-grained Contrastive Learning

TL;DR

This paper introduces CVSA, a contrastive learning framework that enhances fine-grained visual representation by improving object localization through saliency region manipulation and cross-view alignment.

Contribution

The paper proposes a novel saliency-based view generation method and a cross-view alignment loss to improve localization in self-supervised fine-grained learning.

Findings

CVSA significantly improves fine-grained classification accuracy.

The method enhances the model's ability to localize foreground objects.

Experiments show superior performance on multiple benchmarks.

Abstract

Self-supervised contrastive learning has demonstrated great potential in learning visual representations. Despite their success in various downstream tasks such as image classification and object detection, self-supervised pre-training for fine-grained scenarios is not fully explored. We point out that current contrastive methods are prone to memorizing background/foreground texture and therefore have a limitation in localizing the foreground object. Analysis suggests that learning to extract discriminative texture information and localization are equally crucial for fine-grained self-supervised pre-training. Based on our findings, we introduce cross-view saliency alignment (CVSA), a contrastive learning framework that first crops and swaps saliency regions of images as a novel view generation and then guides the model to localize on foreground objects via a cross-view alignment loss. Extensive experiments on both small- and large-scale fine-grained classification benchmarks show that CVSA significantly improves the learned representation.