Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution

TL;DR

This paper introduces LAMNet, a lightweight convolution-based Transformer for image super-resolution that combines adaptive spatial aggregation with efficient design, achieving high performance with significantly reduced inference latency.

Contribution

The paper proposes a novel convolution-based linear focal separable attention mechanism and a dual-branch structure with an information exchange module, enabling efficient long-range modeling in super-resolution.

Findings

LAMNet outperforms existing SA-based Transformers in accuracy.

LAMNet achieves a 3x speedup in inference time.

The proposed method maintains high performance with lower computational cost.

Abstract

Window-based transformers have demonstrated outstanding performance in super-resolution tasks due to their adaptive modeling capabilities through local self-attention (SA). However, they exhibit higher computational complexity and inference latency than convolutional neural networks. In this paper, we first identify that the adaptability of the Transformers is derived from their adaptive spatial aggregation and advanced structural design, while their high latency results from the computational costs and memory layout transformations associated with the local SA. To simulate this aggregation approach, we propose an effective convolution-based linear focal separable attention (FSA), allowing for long-range dynamic modeling with linear complexity. Additionally, we introduce an effective dual-branch structure combined with an ultra-lightweight information exchange module (IEM) to enhance the aggregation of information by the Token Mixer. Finally, with respect to the structure, we modify the existing spatial-gate-based feedforward neural networks by incorporating a self-gate mechanism to preserve high-dimensional channel information, enabling the modeling of more complex relationships. With these advancements, we construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet). Extensive experiments demonstrate that LAMNet achieves better performance than existing SA-based Transformer methods while maintaining the computational efficiency of convolutional neural networks, which can achieve a \(3\times\) speedup of inference time. The code will be publicly available at: https://github.com/zononhzy/LAMNet.