VLM-Augmented Degradation Modeling for Image Restoration Under Adverse Weather Conditions

TL;DR

This paper introduces MVLR, a novel model combining visual-language reasoning and memory modules to improve image restoration under various severe weather conditions, achieving superior results efficiently.

Contribution

The paper presents a unified MVLR model that integrates a visual-language model with an implicit memory bank for adaptive weather degradation priors, enhancing restoration performance.

Findings

MVLR outperforms baselines in PSNR and SSIM on four weather benchmarks.

The model achieves a good balance between accuracy and computational efficiency.

MVLR demonstrates robustness across diverse severe weather conditions.

Abstract

Reliable visual perception under adverse weather conditions, such as rain, haze, snow, or a mixture of them, is desirable yet challenging for autonomous driving and outdoor robots. In this paper, we propose a unified Memory-Enhanced Visual-Language Recovery (MVLR) model that restores images from different degradation levels under various weather conditions. MVLR couples a lightweight encoder-decoder backbone with a Visual-Language Model (VLM) and an Implicit Memory Bank (IMB). The VLM performs chain-of-thought inference to encode weather degradation priors and the IMB stores continuous latent representations of degradation patterns. The VLM-generated priors query the IMB to retrieve fine-grained degradation prototypes. These prototypes are then adaptively fused with multi-scale visual features via dynamic cross-attention mechanisms, enhancing restoration accuracy while maintaining computational efficiency. Extensive experiments on four severe-weather benchmarks show that MVLR surpasses single-branch and Mixture-of-Experts baselines in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These results indicate that MVLR offers a practical balance between model compactness and expressiveness for real-time deployment in diverse outdoor conditions.