Reciprocal Attention Mixing Transformer for Lightweight Image Restoration

TL;DR

This paper introduces RAMiT, a lightweight image restoration network using reciprocal attention mixing in a Transformer architecture, achieving state-of-the-art results across multiple IR tasks with fewer parameters.

Contribution

The paper proposes a novel reciprocal attention mixing Transformer with bi-dimensional self-attentions and hierarchical modules, enhancing efficiency and performance in lightweight IR models.

Findings

Achieves state-of-the-art performance on multiple IR tasks

Reduces model parameters while maintaining high accuracy

Demonstrates effectiveness of bi-dimensional attention in IR

Abstract

Although many recent works have made advancements in the image restoration (IR) field, they often suffer from an excessive number of parameters. Another issue is that most Transformer-based IR methods focus only on either local or global features, leading to limited receptive fields or deficient parameter issues. To address these problems, we propose a lightweight IR network, Reciprocal Attention Mixing Transformer (RAMiT). It employs our proposed dimensional reciprocal attention mixing Transformer (D-RAMiT) blocks, which compute bi-dimensional (spatial and channel) self-attentions in parallel with different numbers of multi-heads. The bi-dimensional attentions help each other to complement their counterpart's drawbacks and are then mixed. Additionally, we introduce a hierarchical reciprocal attention mixing (H-RAMi) layer that compensates for pixel-level information losses and utilizes semantic information while maintaining an efficient hierarchical structure. Furthermore, we revisit and modify MobileNet V1 and V2 to attach efficient convolutions to our proposed components. The experimental results demonstrate that RAMiT achieves state-of-the-art performance on multiple lightweight IR tasks, including super-resolution, color denoising, grayscale denoising, low-light enhancement, and deraining. Codes are available at https://github.com/rami0205/RAMiT.