Masked Frequency Modeling for Self-Supervised Visual Pre-Training

TL;DR

This paper introduces Masked Frequency Modeling (MFM), a novel self-supervised pre-training method that masks and predicts frequency components of images, leading to improved visual representations without extra data or model complexity.

Contribution

MFM is the first to apply frequency domain masking for self-supervised visual pre-training, demonstrating its effectiveness across various models and tasks.

Findings

MFM achieves competitive image classification and segmentation results.

MFM enhances robustness against various image corruptions.

Frequency-based masking reveals underlying image patterns more effectively.

Abstract

We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models. Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain. Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy. Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token. Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches. Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach.