Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal

TL;DR

This paper investigates leveraging large-scale pre-trained models like CLIP for adverse weather removal in images, introducing novel modules and strategies to improve spatial feature extraction and semantic understanding, achieving state-of-the-art results.

Contribution

It proposes a new framework utilizing CLIP for weather removal, including SAR encoder, CLIP-SRD distillation, and CWP embedding, which are novel integrations for this task.

Findings

Achieves state-of-the-art performance on adverse weather removal benchmarks.

Demonstrates the effectiveness of CLIP-based semantic and spatial feature integration.

Outperforms existing methods in challenging weather conditions.

Abstract

Image restoration under adverse weather conditions (e.g., rain, snow and haze) is a fundamental computer vision problem and has important indications for various downstream applications. Different from early methods that are specially designed for specific type of weather, most recent works tend to remove various adverse weather effects simultaneously through either spatial feature representation learning or semantic information embedding. Inspired by the various successful applications of large-scale pre-trained models (e.g, CLIP), in this paper, we explore the potential benefits of them for this task through both spatial feature representation learning and semantic information embedding aspects: 1) for spatial feature representation learning, we design a Spatially-Adaptive Residual (\textbf{SAR}) Encoder to extract degraded areas adaptively. To facilitate its training, we propose a Soft Residual Distillation (\textbf{CLIP-SRD}) strategy to transfer the spatial knowledge from CLIP between clean and adverse weather images; 2) for semantic information embedding, we propose a CLIP Weather Prior (\textbf{CWP}) embedding module to make the network handle different weather conditions adaptively. This module integrates the sample specific weather prior extracted by CLIP image encoder together with the distribution specific information learned by a set of parameters, and embeds them through a cross attention mechanism. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art performance under different and challenging adverse weather conditions. Code will be made available.