Breaking Complexity Barriers: High-Resolution Image Restoration with Rank Enhanced Linear Attention

TL;DR

This paper introduces LAformer, a high-resolution image restoration model that uses Rank Enhanced Linear Attention to efficiently capture global context, outperforming state-of-the-art methods while reducing computational complexity.

Contribution

The paper proposes RELA, a novel rank enhancement for linear attention, and integrates it into LAformer, an efficient Transformer for high-resolution image restoration.

Findings

LAformer outperforms SOTA methods across 7 IR tasks and 21 benchmarks.

LAformer achieves significant computational efficiency improvements.

RELA effectively enriches feature representations, improving IR performance.

Abstract

Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Transformer hinders its applicability to high-resolution images. Existing methods mitigate this issue with sparse or window-based attention, yet inherently limit global context modeling. Linear attention, a variant of softmax attention, demonstrates promise in global context modeling while maintaining linear complexity, offering a potential solution to the above challenge. Despite its efficiency benefits, vanilla linear attention suffers from a significant performance drop in IR, largely due to the low-rank nature of its attention map. To counter this, we propose Rank Enhanced Linear Attention (RELA), a simple yet effective method that enriches feature representations by integrating a lightweight depthwise convolution. Building upon RELA, we propose an efficient and effective image restoration Transformer, named LAformer. LAformer achieves effective global perception by integrating linear attention and channel attention, while also enhancing local fitting capabilities through a convolutional gated feed-forward network. Notably, LAformer eliminates hardware-inefficient operations such as softmax and window shifting, enabling efficient processing of high-resolution images. Extensive experiments across 7 IR tasks and 21 benchmarks demonstrate that LAformer outperforms SOTA methods and offers significant computational advantages.