A Hybrid Transformer-Mamba Network for Single Image Deraining

TL;DR

This paper introduces TransMamba, a hybrid Transformer-Mamba network that effectively captures long-range dependencies and spectral information for single image deraining, outperforming existing methods.

Contribution

The paper proposes a dual-branch hybrid network with spectral-banded Transformer and Mamba layers, incorporating spectral coherence loss for improved deraining performance.

Findings

Outperforms state-of-the-art deraining methods on multiple datasets.

Effectively captures long-range and spectral dependencies in images.

Demonstrates superior reconstruction of clean images with preserved details.

Abstract

Existing deraining Transformers employ self-attention mechanisms with fixed-range windows or along channel dimensions, limiting the exploitation of non-local receptive fields. In response to this issue, we introduce a novel dual-branch hybrid Transformer-Mamba network, denoted as TransMamba, aimed at effectively capturing long-range rain-related dependencies. Based on the prior of distinct spectral-domain features of rain degradation and background, we design a spectral-banded Transformer blocks on the first branch. Self-attention is executed within the combination of the spectral-domain channel dimension to improve the ability of modeling long-range dependencies. To enhance frequency-specific information, we present a spectral enhanced feed-forward module that aggregates features in the spectral domain. In the second branch, Mamba layers are equipped with cascaded bidirectional state space model modules to additionally capture the modeling of both local and global information. At each stage of both the encoder and decoder, we perform channel-wise concatenation of dual-branch features and achieve feature fusion through channel reduction, enabling more effective integration of the multi-scale information from the Transformer and Mamba branches. To better reconstruct innate signal-level relations within clean images, we also develop a spectral coherence loss. Extensive experiments on diverse datasets and real-world images demonstrate the superiority of our method compared against the state-of-the-art approaches.