Learning Texture Transformer Network for Image Super-Resolution

TL;DR

This paper introduces a Texture Transformer Network for image super-resolution that leverages attention mechanisms to transfer high-resolution textures from reference images, significantly improving texture recovery and image quality.

Contribution

The paper proposes a novel transformer-based architecture with modules for texture extraction, relevance embedding, and attention-based texture transfer and synthesis for super-resolution.

Findings

Achieves significant improvements over state-of-the-art methods.

Effectively transfers textures from reference images across multiple scales.

Demonstrates superior qualitative and quantitative results.

Abstract

We study on image super-resolution (SR), which aims to recover realistic textures from a low-resolution (LR) image. Recent progress has been made by taking high-resolution images as references (Ref), so that relevant textures can be transferred to LR images. However, existing SR approaches neglect to use attention mechanisms to transfer high-resolution (HR) textures from Ref images, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively. TTSR consists of four closely-related modules optimized for image generation tasks, including a learnable texture extractor by DNN, a relevance embedding module, a hard-attention module for texture transfer, and a soft-attention module for texture synthesis. Such a design encourages joint feature learning across LR and Ref images, in which deep feature correspondences can be discovered by attention, and thus accurate texture features can be transferred. The proposed texture transformer can be further stacked in a cross-scale way, which enables texture recovery from different levels (e.g., from 1x to 4x magnification). Extensive experiments show that TTSR achieves significant improvements over state-of-the-art approaches on both quantitative and qualitative evaluations.