Symmetric masking strategy enhances the performance of Masked Image Modeling

TL;DR

This paper introduces SymMIM, a symmetric masking strategy for Masked Image Modeling that improves global and local feature learning, leading to state-of-the-art results on ImageNet and various vision tasks.

Contribution

The paper proposes a novel symmetric masking strategy for MIM that reduces resource requirements and enhances model performance across multiple vision benchmarks.

Findings

Achieves 85.9% accuracy on ImageNet with ViT-Large

Surpasses previous SOTA on downstream vision tasks

Reduces pre-training epochs needed for optimal performance

Abstract

Masked Image Modeling (MIM) is a technique in self-supervised learning that focuses on acquiring detailed visual representations from unlabeled images by estimating the missing pixels in randomly masked sections. It has proven to be a powerful tool for the preliminary training of Vision Transformers (ViTs), yielding impressive results across various tasks. Nevertheless, most MIM methods heavily depend on the random masking strategy to formulate the pretext task. This strategy necessitates numerous trials to ascertain the optimal dropping ratio, which can be resource-intensive, requiring the model to be pre-trained for anywhere between 800 to 1600 epochs. Furthermore, this approach may not be suitable for all datasets. In this work, we propose a new masking strategy that effectively helps the model capture global and local features. Based on this masking strategy, SymMIM, our proposed training pipeline for MIM is introduced. SymMIM achieves a new SOTA accuracy of 85.9\% on ImageNet using ViT-Large and surpasses previous SOTA across downstream tasks such as image classification, semantic segmentation, object detection, instance segmentation tasks, and so on.