Pre-training with Random Orthogonal Projection Image Modeling

TL;DR

This paper introduces ROPIM, a novel self-supervised image pre-training method using random orthogonal projections instead of traditional masking, achieving superior results on benchmarks.

Contribution

Proposes ROPIM, a new masking approach based on orthogonal projections, improving over crop-based masking in masked image modeling.

Findings

ROPIM outperforms crop-based masking in experiments.

Achieves state-of-the-art results on multiple benchmarks.

Reduces spatial information with controlled noise variance.

Abstract

Masked Image Modeling (MIM) is a powerful self-supervised strategy for visual pre-training without the use of labels. MIM applies random crops to input images, processes them with an encoder, and then recovers the masked inputs with a decoder, which encourages the network to capture and learn structural information about objects and scenes. The intermediate feature representations obtained from MIM are suitable for fine-tuning on downstream tasks. In this paper, we propose an Image Modeling framework based on random orthogonal projection instead of binary masking as in MIM. Our proposed Random Orthogonal Projection Image Modeling (ROPIM) reduces spatially-wise token information under guaranteed bound on the noise variance and can be considered as masking entire spatial image area under locally varying masking degrees. Since ROPIM uses a random subspace for the projection that realizes the masking step, the readily available complement of the subspace can be used during unmasking to promote recovery of removed information. In this paper, we show that using random orthogonal projection leads to superior performance compared to crop-based masking. We demonstrate state-of-the-art results on several popular benchmarks.