Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where

TL;DR

This paper enhances contrastive self-supervised learning for convolutional neural networks by incorporating saliency-aware masking, leading to more effective data augmentation and improved downstream task performance.

Contribution

It introduces a saliency-constrained masking method for contrastive learning in ConvNets, addressing issues of biased masking and adding hard negatives for better representation learning.

Findings

Saliency-aware masking improves contrastive learning performance.

Evenly distributed masking reduces misleading contrastiveness.

Hard negatives with larger salient masks enhance downstream accuracy.

Abstract

While image data starts to enjoy the simple-but-effective self-supervised learning scheme built upon masking and self-reconstruction objective thanks to the introduction of tokenization procedure and vision transformer backbone, convolutional neural networks as another important and widely-adopted architecture for image data, though having contrastive-learning techniques to drive the self-supervised learning, still face the difficulty of leveraging such straightforward and general masking operation to benefit their learning process significantly. In this work, we aim to alleviate the burden of including masking operation into the contrastive-learning framework for convolutional neural networks as an extra augmentation method. In addition to the additive but unwanted edges (between masked and unmasked regions) as well as other adverse effects caused by the masking operations for ConvNets, which have been discussed by prior works, we particularly identify the potential problem where for one view in a contrastive sample-pair the randomly-sampled masking regions could be overly concentrated on important/salient objects thus resulting in misleading contrastiveness to the other view. To this end, we propose to explicitly take the saliency constraint into consideration in which the masked regions are more evenly distributed among the foreground and background for realizing the masking-based augmentation. Moreover, we introduce hard negative samples by masking larger regions of salient patches in an input image. Extensive experiments conducted on various datasets, contrastive learning mechanisms, and downstream tasks well verify the efficacy as well as the superior performance of our proposed method with respect to several state-of-the-art baselines.