Stare at What You See: Masked Image Modeling without Reconstruction

TL;DR

MaskAlign introduces a reconstruction-free masked image modeling approach that leverages feature consistency with a teacher model, achieving state-of-the-art results with higher efficiency.

Contribution

The paper proposes MaskAlign, a novel masked image modeling method that eliminates the need for reconstruction by aligning features from student and teacher models.

Findings

Reconstruction is not necessary for effective masked image modeling.

MaskAlign achieves state-of-the-art performance on vision benchmarks.

Dynamic Alignment improves feature consistency and model performance.

Abstract

Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pre-training. By reconstructing masked image patches from a small portion of visible image regions, MAE forces the model to infer semantic correlation within an image. Recently, some approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image.This raises one question: is reconstruction necessary in Masked Image Modeling (MIM) with a teacher model? In this paper, we propose an efficient MIM paradigm named MaskAlign. MaskAlign simply learns the consistency of visible patch features extracted by the student model and intact image features extracted by the teacher model. To further advance the performance and tackle the problem of input inconsistency between the student and teacher model, we propose a Dynamic Alignment (DA) module to apply learnable alignment. Our experimental results demonstrate that masked modeling does not lose effectiveness even without reconstruction on masked regions. Combined with Dynamic Alignment, MaskAlign can achieve state-of-the-art performance with much higher efficiency. Code and models will be available at https://github.com/OpenPerceptionX/maskalign.