Improving Generative Pre-Training: An In-depth Study of Masked Image Modeling and Denoising Models

TL;DR

This paper investigates the effects of additive noise in masked image modeling and proposes specific conditions for effective integration, leading to improved pre-training performance on recognition tasks.

Contribution

It identifies key conditions for combining noise with masked image modeling and demonstrates enhanced pre-training results across various recognition tasks.

Findings

Effective noise application within the encoder improves recognition performance.

Feature space noise and token disentanglement are critical for success.

The proposed method enhances fine-grained and high-frequency recognition tasks.

Abstract

In this work, we dive deep into the impact of additive noise in pre-training deep networks. While various methods have attempted to use additive noise inspired by the success of latent denoising diffusion models, when used in combination with masked image modeling, their gains have been marginal when it comes to recognition tasks. We thus investigate why this would be the case, in an attempt to find effective ways to combine the two ideas. Specifically, we find three critical conditions: corruption and restoration must be applied within the encoder, noise must be introduced in the feature space, and an explicit disentanglement between noised and masked tokens is necessary. By implementing these findings, we demonstrate improved pre-training performance for a wide range of recognition tasks, including those that require fine-grained, high-frequency information to solve.