MixMask: Revisiting Masking Strategy for Siamese ConvNets

TL;DR

MixMask introduces a novel filling-based masking strategy for Siamese ConvNets that improves training efficiency and performance across multiple tasks by replacing erased regions with content from other images and employing an adaptive loss.

Contribution

This work proposes MixMask, a filling-based masking approach with an adaptive loss for Siamese ConvNets, addressing limitations of erase-based masking and enhancing self-supervised learning.

Findings

Outperforms MSCN in various tasks

Enhances linear probing, semi-supervised, and supervised finetuning

Improves object detection and segmentation results

Abstract

The recent progress in self-supervised learning has successfully combined Masked Image Modeling (MIM) with Siamese Networks, harnessing the strengths of both methodologies. Nonetheless, certain challenges persist when integrating conventional erase-based masking within Siamese ConvNets. Two primary concerns are: (1) The continuous data processing nature of ConvNets, which doesn't allow for the exclusion of non-informative masked regions, leading to reduced training efficiency compared to ViT architecture; (2) The misalignment between erase-based masking and the contrastive-based objective, distinguishing it from the MIM technique. To address these challenges, this work introduces a novel filling-based masking approach, termed \textbf{MixMask}. The proposed method replaces erased areas with content from a different image, effectively countering the information depletion seen in traditional masking methods. Additionally, we unveil an adaptive loss function that captures the semantics of the newly patched views, ensuring seamless integration within the architectural framework. We empirically validate the effectiveness of our approach through comprehensive experiments across various datasets and application scenarios. The findings underscore our framework's enhanced performance in areas such as linear probing, semi-supervised and supervised finetuning, object detection and segmentation. Notably, our method surpasses the MSCN, establishing MixMask as a more advantageous masking solution for Siamese ConvNets. Our code and models are publicly available at https://github.com/kirill-vish/MixMask.