A Semantic Decoupling-Based Two-Stage Rainy-Day Attack for Revealing Weather Robustness Deficiencies in Vision-Language Models

TL;DR

This paper introduces a novel two-stage adversarial framework that uses realistic rainy weather perturbations to reveal vulnerabilities in vision-language models, highlighting potential safety risks in real-world scenarios.

Contribution

It is the first to systematically analyze rain-induced semantic shifts in VLMs using a physically grounded, two-stage, semantic decoupling-based adversarial approach.

Findings

Rainy weather perturbations cause significant semantic misalignment in VLMs.

Illumination changes and multi-scale raindrop structures are key factors in semantic shifts.

The framework demonstrates potential safety and reliability risks in real-world applications.

Abstract

Vision-Language Models (VLMs) are trained on image-text pairs collected under canonical visual conditions and achieve strong performance on multimodal tasks. However, their robustness to real-world weather conditions, and the stability of cross-modal semantic alignment under such structured perturbations, remain insufficiently studied. In this paper, we focus on rainy scenarios and introduce the first adversarial framework that exploits realistic weather to attack VLMs, using a two-stage, parameterized perturbation model based on semantic decoupling to analyze rain-induced shifts in decision-making. In Stage 1, we model the global effects of rainfall by applying a low-dimensional global modulation to condition the embedding space and gradually weaken the original semantic decision boundaries. In Stage 2, we introduce structured rain variations by explicitly modeling multi-scale raindrop appearance and rainfall-induced illumination changes, and optimize the resulting non-differentiable weather space to induce stable semantic shifts. Operating in a non-pixel parameter space, our framework generates perturbations that are both physically grounded and interpretable. Experiments across multiple tasks show that even physically plausible, highly constrained weather perturbations can induce substantial semantic misalignment in mainstream VLMs, posing potential safety and reliability risks in real-world deployment. Ablations further confirm that illumination modeling and multi-scale raindrop structures are key drivers of these semantic shifts.