Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion

TL;DR

This paper introduces Tmamba, a dual-branch image fusion network combining linear Transformer and Mamba to effectively extract and fuse multi-modality images with linear complexity, outperforming existing methods.

Contribution

The paper proposes a novel dual-branch network with cross-modal attention and interaction mechanisms, enhancing feature extraction and fusion for multi-modality images.

Findings

Achieves promising results in infrared-visible image fusion

Effective in medical image fusion tasks

Maintains linear computational complexity

Abstract

Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process.