MatIR: A Hybrid Mamba-Transformer Image Restoration Model

TL;DR

MatIR is a novel hybrid image restoration model combining Mamba and Transformer architectures to leverage their respective strengths, achieving improved contextual learning and efficiency in image restoration tasks.

Contribution

This paper introduces MatIR, a hybrid Mamba-Transformer model with cross-cycled blocks and novel modules, enhancing image restoration performance beyond existing models.

Findings

Outperforms existing models in image restoration tasks

Effectively captures long-range dependencies with IRSS module

Demonstrates superior efficiency and accuracy in experiments

Abstract

In recent years, Transformers-based models have made significant progress in the field of image restoration by leveraging their inherent ability to capture complex contextual features. Recently, Mamba models have made a splash in the field of computer vision due to their ability to handle long-range dependencies and their significant computational efficiency compared to Transformers. However, Mamba currently lags behind Transformers in contextual learning capabilities. To overcome the limitations of these two models, we propose a Mamba-Transformer hybrid image restoration model called MatIR. Specifically, MatIR cross-cycles the blocks of the Transformer layer and the Mamba layer to extract features, thereby taking full advantage of the advantages of the two architectures. In the Mamba module, we introduce the Image Inpainting State Space (IRSS) module, which traverses along four scan paths to achieve efficient processing of long sequence data. In the Transformer module, we combine triangular window-based local attention with channel-based global attention to effectively activate the attention mechanism over a wider range of image pixels. Extensive experimental results and ablation studies demonstrate the effectiveness of our approach.