LMLT: Low-to-high Multi-Level Vision Transformer for Image Super-Resolution

TL;DR

LMLT introduces a multi-level vision transformer that efficiently captures local and global features for image super-resolution, reducing complexity and memory usage while maintaining or surpassing state-of-the-art performance.

Contribution

The paper proposes a novel multi-level transformer architecture that addresses window boundary issues and reduces computational complexity in image super-resolution tasks.

Findings

Significantly reduces inference time and GPU memory usage.

Maintains or exceeds the performance of existing ViT-based methods.

Effectively captures both local and global image features.

Abstract

Recent Vision Transformer (ViT)-based methods for Image Super-Resolution have demonstrated impressive performance. However, they suffer from significant complexity, resulting in high inference times and memory usage. Additionally, ViT models using Window Self-Attention (WSA) face challenges in processing regions outside their windows. To address these issues, we propose the Low-to-high Multi-Level Transformer (LMLT), which employs attention with varying feature sizes for each head. LMLT divides image features along the channel dimension, gradually reduces spatial size for lower heads, and applies self-attention to each head. This approach effectively captures both local and global information. By integrating the results from lower heads into higher heads, LMLT overcomes the window boundary issues in self-attention. Extensive experiments show that our model significantly reduces inference time and GPU memory usage while maintaining or even surpassing the performance of state-of-the-art ViT-based Image Super-Resolution methods. Our codes are availiable at https://github.com/jwgdmkj/LMLT.