HiTSR: A Hierarchical Transformer for Reference-based Super-Resolution

TL;DR

HiTSR introduces a hierarchical transformer that effectively enhances low-resolution images using reference images, achieving state-of-the-art super-resolution results with a streamlined architecture and attention mechanisms.

Contribution

The paper presents a novel hierarchical transformer model for reference-based super-resolution that simplifies existing architectures and leverages attention mechanisms for improved performance.

Findings

Achieves state-of-the-art PSNR/SSIM on SUN80 dataset.

Outperforms existing methods across multiple datasets.

Utilizes attention mechanisms effectively without complex subnetworks.

Abstract

In this paper, we propose HiTSR, a hierarchical transformer model for reference-based image super-resolution, which enhances low-resolution input images by learning matching correspondences from high-resolution reference images. Diverging from existing multi-network, multi-stage approaches, we streamline the architecture and training pipeline by incorporating the double attention block from GAN literature. Processing two visual streams independently, we fuse self-attention and cross-attention blocks through a gating attention strategy. The model integrates a squeeze-and-excitation module to capture global context from the input images, facilitating long-range spatial interactions within window-based attention blocks. Long skip connections between shallow and deep layers further enhance information flow. Our model demonstrates superior performance across three datasets including SUN80, Urban100, and Manga109. Specifically, on the SUN80 dataset, our model achieves PSNR/SSIM values of 30.24/0.821. These results underscore the effectiveness of attention mechanisms in reference-based image super-resolution. The transformer-based model attains state-of-the-art results without the need for purpose-built subnetworks, knowledge distillation, or multi-stage training, emphasizing the potency of attention in meeting reference-based image super-resolution requirements.