Mixed Autoencoder for Self-supervised Visual Representation Learning

TL;DR

This paper introduces Mixed Autoencoder (MixedAE), a novel self-supervised learning method that uses mixing augmentation with an auxiliary recognition task to improve visual representation learning and outperform existing methods.

Contribution

It proposes homologous recognition as a new auxiliary task to effectively utilize mixing augmentation in masked image modeling, achieving state-of-the-art results.

Findings

MixedAE outperforms MAE on multiple downstream tasks.

Homologous recognition alleviates mutual information increase.

MixedAE accelerates training while improving accuracy.

Abstract

Masked Autoencoder (MAE) has demonstrated superior performance on various vision tasks via randomly masking image patches and reconstruction. However, effective data augmentation strategies for MAE still remain open questions, different from those in contrastive learning that serve as the most important part. This paper studies the prevailing mixing augmentation for MAE. We first demonstrate that naive mixing will in contrast degenerate model performance due to the increase of mutual information (MI). To address, we propose homologous recognition, an auxiliary pretext task, not only to alleviate the MI increasement by explicitly requiring each patch to recognize homologous patches, but also to perform object-aware self-supervised pre-training for better downstream dense perception performance. With extensive experiments, we demonstrate that our proposed Mixed Autoencoder (MixedAE) achieves the state-of-the-art transfer results among masked image modeling (MIM) augmentations on different downstream tasks with significant efficiency. Specifically, our MixedAE outperforms MAE by +0.3% accuracy, +1.7 mIoU and +0.9 AP on ImageNet-1K, ADE20K and COCO respectively with a standard ViT-Base. Moreover, MixedAE surpasses iBOT, a strong MIM method combined with instance discrimination, while accelerating training by 2x. To our best knowledge, this is the very first work to consider mixing for MIM from the perspective of pretext task design. Code will be made available.