# Effective and efficient self-supervised masked model based on mixed feature training

## TL;DR

This paper introduces EESMM, a self-supervised masked model that reduces pre-training time while maintaining accuracy by using mixed feature training.

## Contribution

EESMM introduces a novel mixed feature training approach that reduces computational costs and pre-training time significantly.

## Key findings

- EESMM achieves 83% accuracy on ImageNet in 363 hours using four V100 GPUs.
- The method reduces pre-training time to one-tenth of SimMIM without performance loss.
- Mixed feature training enhances feature representation and reduces computational complexity.

## Abstract

Under the influence of Masked Language Modeling (MLM), Masked Image Modeling (MIM) employs an attention mechanism to perform masked training on images. However, processing a single image requires numerous iterations and substantial computational resources to reconstruct the masked regions, resulting in high computational complexity and significant time costs. To address this issue, we propose an Effective and Efficient self-supervised Masked model based on Mixed feature training (EESMM). First, we stack two images for encoding and input the fused features into the network, which not only reduces computational complexity but also enables the learning of more features. Second, during decoding, we obtain the decoding features corresponding to the original images based on the decoding features of the two input original images and the mixed images, and then construct a corresponding loss function to enhance feature representation. EESMM significantly reduces pre-training time without sacrificing accuracy, achieving 83% accuracy on ImageNet in just 363 h using four V100 GPUs–only one-tenth of the training time required by SimMIM. This validates that the method can substantially accelerate the pre-training process without noticeable performance degradation.