PTSR: Patch Translator for Image Super-Resolution

TL;DR

This paper introduces PTSR, a transformer-based GAN model without convolution, that improves image super-resolution quality by regenerating patches with multi-head attention, achieving significant performance gains on benchmark datasets.

Contribution

The paper presents a novel convolution-free transformer GAN with a patch translator module for enhanced super-resolution image generation.

Findings

Achieves 21.66% improvement in PSNR for 4x super-resolution

Achieves 11.59% improvement in SSIM for 4x super-resolution

Effective patch regeneration demonstrated through saliency map analysis

Abstract

Image super-resolution generation aims to generate a high-resolution image from its low-resolution image. However, more complex neural networks bring high computational costs and memory storage. It is still an active area for offering the promise of overcoming resolution limitations in many applications. In recent years, transformers have made significant progress in computer vision tasks as their robust self-attention mechanism. However, recent works on the transformer for image super-resolution also contain convolution operations. We propose a patch translator for image super-resolution (PTSR) to address this problem. The proposed PTSR is a transformer-based GAN network with no convolution operation. We introduce a novel patch translator module for regenerating the improved patches utilising multi-head attention, which is further utilised by the generator to generate the 2x and 4x super-resolution images. The experiments are performed using benchmark datasets, including DIV2K, Set5, Set14, and BSD100. The results of the proposed model is improved on an average for $4\times$ super-resolution by 21.66% in PNSR score and 11.59% in SSIM score, as compared to the best competitive models. We also analyse the proposed loss and saliency map to show the effectiveness of the proposed method.