RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining

TL;DR

RainMamba introduces a novel SSMs-based video deraining network utilizing a Hilbert scanning mechanism and contrastive locality learning, achieving efficient and effective rain removal in videos with improved local and temporal feature modeling.

Contribution

The paper proposes an improved SSMs-based network with a Hilbert scanning mechanism and contrastive locality learning for better local and temporal feature extraction in video deraining.

Findings

Effective rain streak and raindrop removal demonstrated on multiple datasets.

Outperforms existing methods in efficiency and accuracy.

Code and results publicly available for reproducibility.

Abstract

The outdoor vision systems are frequently contaminated by rain streaks and raindrops, which significantly degenerate the performance of visual tasks and multimedia applications. The nature of videos exhibits redundant temporal cues for rain removal with higher stability. Traditional video deraining methods heavily rely on optical flow estimation and kernel-based manners, which have a limited receptive field. Yet, transformer architectures, while enabling long-term dependencies, bring about a significant increase in computational complexity. Recently, the linear-complexity operator of the state space models (SSMs) has contrarily facilitated efficient long-term temporal modeling, which is crucial for rain streaks and raindrops removal in videos. Unexpectedly, its uni-dimensional sequential process on videos destroys the local correlations across the spatio-temporal dimension by distancing adjacent pixels. To address this, we present an improved SSMs-based video deraining network (RainMamba) with a novel Hilbert scanning mechanism to better capture sequence-level local information. We also introduce a difference-guided dynamic contrastive locality learning strategy to enhance the patch-level self-similarity learning ability of the proposed network. Extensive experiments on four synthesized video deraining datasets and real-world rainy videos demonstrate the effectiveness and efficiency of our network in the removal of rain streaks and raindrops. Our code and results are available at https://github.com/TonyHongtaoWu/RainMamba.