On Efficient Transformer-Based Image Pre-training for Low-Level Vision

TL;DR

This paper investigates how transformer-based pre-training influences low-level vision tasks, revealing task-specific effects and proposing effective multi-task pre-training strategies to achieve state-of-the-art results.

Contribution

It provides a comprehensive analysis of pre-training effects on low-level vision tasks and introduces effective multi-task pre-training methods for transformers.

Findings

Pre-training introduces more local information in super-resolution.

Pre-training has limited impact on denoising internal features.

Multi-related-task pre-training outperforms other methods.

Abstract

Pre-training has marked numerous state of the arts in high-level computer vision, while few attempts have ever been made to investigate how pre-training acts in image processing systems. In this paper, we tailor transformer-based pre-training regimes that boost various low-level tasks. To comprehensively diagnose the influence of pre-training, we design a whole set of principled evaluation tools that uncover its effects on internal representations. The observations demonstrate that pre-training plays strikingly different roles in low-level tasks. For example, pre-training introduces more local information to higher layers in super-resolution (SR), yielding significant performance gains, while pre-training hardly affects internal feature representations in denoising, resulting in limited gains. Further, we explore different methods of pre-training, revealing that multi-related-task pre-training is more effective and data-efficient than other alternatives. Finally, we extend our study to varying data scales and model sizes, as well as comparisons between transformers and CNNs-based architectures. Based on the study, we successfully develop state-of-the-art models for multiple low-level tasks. Code is released at https://github.com/fenglinglwb/EDT.