Energy-Inspired Self-Supervised Pretraining for Vision Models

TL;DR

This paper introduces an energy-based, self-supervised pretraining framework for vision models that unifies data encoding and restoration in a single network, enabling diverse pretext tasks and achieving competitive results with fewer training epochs.

Contribution

It proposes a novel energy-inspired pretraining method that integrates forward and backward passes into one model, eliminating the need for auxiliary components and broadening pretext task options.

Findings

Achieves comparable or better performance than state-of-the-art methods.

Requires significantly fewer training epochs.

Supports various pretext tasks like masking, denoising, and super-resolution.

Abstract

Motivated by the fact that forward and backward passes of a deep network naturally form symmetric mappings between input and output representations, we introduce a simple yet effective self-supervised vision model pretraining framework inspired by energy-based models (EBMs). In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network without any auxiliary components, e.g., an extra decoder. For the forward pass, we fit a network to an energy function that assigns low energy scores to samples that belong to an unlabeled dataset, and high energy otherwise. For the backward pass, we restore data from corrupted versions iteratively using gradient-based optimization along the direction of energy minimization. In this way, we naturally fold the encoder-decoder architecture widely used in masked image modeling into the forward and backward passes of a single vision model. Thus, our framework now accepts a wide range of pretext tasks with different data corruption methods, and permits models to be pretrained from masked image modeling, patch sorting, and image restoration, including super-resolution, denoising, and colorization. We support our findings with extensive experiments, and show the proposed method delivers comparable and even better performance with remarkably fewer epochs of training compared to the state-of-the-art self-supervised vision model pretraining methods. Our findings shed light on further exploring self-supervised vision model pretraining and pretext tasks beyond masked image modeling.