CLIP-driven rain perception: Adaptive deraining with pattern-aware network routing and mask-guided cross-attention

TL;DR

This paper introduces CLIP-RPN, a rain deraining model that uses CLIP for rain pattern recognition and adaptive routing, combined with a mask-guided cross-attention mechanism and dynamic loss scheduling, to effectively handle diverse rain conditions.

Contribution

The paper presents a novel CLIP-driven rain perception network with adaptive sub-network routing and a mask-guided cross-attention mechanism, improving deraining performance across varied rain patterns.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively handles complex mixed rain scenarios.

Outperforms traditional methods using fixed loss functions.

Abstract

Existing deraining models process all rainy images within a single network. However, different rain patterns have significant variations, which makes it challenging for a single network to handle diverse types of raindrops and streaks. To address this limitation, we propose a novel CLIP-driven rain perception network (CLIP-RPN) that leverages CLIP to automatically perceive rain patterns by computing visual-language matching scores and adaptively routing to sub-networks to handle different rain patterns, such as varying raindrop densities, streak orientations, and rainfall intensity. CLIP-RPN establishes semantic-aware rain pattern recognition through CLIP's cross-modal visual-language alignment capabilities, enabling automatic identification of precipitation characteristics across different rain scenarios. This rain pattern awareness drives an adaptive subnetwork routing mechanism where specialized processing branches are dynamically activated based on the detected rain type, significantly enhancing the model's capacity to handle diverse rainfall conditions. Furthermore, within sub-networks of CLIP-RPN, we introduce a mask-guided cross-attention mechanism (MGCA) that predicts precise rain masks at multi-scale to facilitate contextual interactions between rainy regions and clean background areas by cross-attention. We also introduces a dynamic loss scheduling mechanism (DLS) to adaptively adjust the gradients for the optimization process of CLIP-RPN. Compared with the commonly used $l_1$ or $l_2$ loss, DLS is more compatible with the inherent dynamics of the network training process, thus achieving enhanced outcomes. Our method achieves state-of-the-art performance across multiple datasets, particularly excelling in complex mixed datasets.