Corrupted Image Modeling for Self-Supervised Visual Pre-Training

TL;DR

This paper presents Corrupted Image Modeling (CIM), a novel self-supervised pre-training method that corrupts images with a learnable generator and trains an enhancer to recover or identify corruptions, improving visual representations.

Contribution

CIM introduces a unified, non-Siamese framework for ViT and CNN architectures using a generator-enhancer setup for self-supervised learning.

Findings

Achieves strong results on ImageNet classification.

Effective for ADE20K semantic segmentation.

Compatible with various network architectures.

Abstract

We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial [MASK] tokens, where some patches are randomly selected and replaced with plausible alternatives sampled from the BEiT output distribution. Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not. The generator and the enhancer are simultaneously trained and synergistically updated. After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks. CIM is a general and flexible visual pre-training framework that is suitable for various network architectures. For the first time, CIM demonstrates that both ViT and CNN can learn rich visual representations using a unified, non-Siamese framework. Experimental results show that our approach achieves compelling results in vision benchmarks, such as ImageNet classification and ADE20K semantic segmentation.