Robust Adverse Weather Removal via Spectral-based Spatial Grouping

TL;DR

This paper introduces SSGformer, a spectral-based spatial grouping transformer that effectively removes diverse adverse weather effects from images by decomposing features and applying group-wise attention for robust restoration.

Contribution

The paper proposes a novel spectral decomposition and spatial grouping method with a transformer architecture for improved multi-weather image restoration.

Findings

Outperforms existing methods in adverse weather removal tasks.

Effectively handles highly variable and localized weather degradations.

Demonstrates robustness across diverse weather conditions.

Abstract

Adverse weather conditions cause diverse and complex degradation patterns, driving the development of All-in-One (AiO) models. However, recent AiO solutions still struggle to capture diverse degradations, since global filtering methods like direct operations on the frequency domain fail to handle highly variable and localized distortions. To address these issue, we propose Spectral-based Spatial Grouping Transformer (SSGformer), a novel approach that leverages spectral decomposition and group-wise attention for multi-weather image restoration. SSGformer decomposes images into high-frequency edge features using conventional edge detection and low-frequency information via Singular Value Decomposition. We utilize multi-head linear attention to effectively model the relationship between these features. The fused features are integrated with the input to generate a grouping-mask that clusters regions based on the spatial similarity and image texture. To fully leverage this mask, we introduce a group-wise attention mechanism, enabling robust adverse weather removal and ensuring consistent performance across diverse weather conditions. We also propose a Spatial Grouping Transformer Block that uses both channel attention and spatial attention, effectively balancing feature-wise relationships and spatial dependencies. Extensive experiments show the superiority of our approach, validating its effectiveness in handling the varied and intricate adverse weather degradations.