Pre-Trained Image Processing Transformer

TL;DR

This paper introduces the Image Processing Transformer (IPT), a pre-trained model leveraging transformer architecture and contrastive learning, trained on large-scale corrupted images, to excel at low-level vision tasks like denoising and super-resolution.

Contribution

The paper presents a novel pre-trained transformer model for low-level vision tasks, utilizing large-scale corrupted image data and contrastive learning for effective task adaptation.

Findings

IPT outperforms state-of-the-art methods on multiple benchmarks.

A single pre-trained model effectively adapts to various low-level tasks.

Contrastive learning enhances the model's versatility across different image processing tasks.

Abstract

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT