Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

TL;DR

This paper introduces Multimodal Semantic Lighting Attacks (MSLA), a novel physical adversarial attack method that disrupts vision-language models using controllable lighting, revealing significant vulnerabilities in real-world scenarios.

Contribution

It is the first to systematically study physically deployable semantic attacks on VLMs, demonstrating their effectiveness and exposing a critical robustness gap.

Findings

MSLA significantly degrades zero-shot classification accuracy.

MSLA induces semantic hallucinations in image captioning and VQA.

Physical attacks using lighting are practical and transferable.

Abstract

Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world threats largely unexplored. As VLMs are increasingly deployed in real environments, this gap becomes critical, since adversarial perturbations must be physically realizable. Despite this practical relevance, physical attacks against VLMs have not been systematically studied. Such attacks may induce recognition failures and further disrupt multimodal reasoning, leading to severe semantic misinterpretation in downstream tasks. Therefore, investigating physical attacks on VLMs is essential for assessing their real-world security risks. To address this gap, we propose Multimodal Semantic Lighting Attacks (MSLA), the first physically deployable adversarial attack framework against VLMs. MSLA uses controllable adversarial lighting to disrupt multimodal semantic understanding in real scenes, attacking semantic alignment rather than only task-specific outputs. Consequently, it degrades zero-shot classification performance of mainstream CLIP variants while inducing severe semantic hallucinations in advanced VLMs such as LLaVA and BLIP across image captioning and visual question answering (VQA). Extensive experiments in both digital and physical domains demonstrate that MSLA is effective, transferable, and practically realizable. Our findings provide the first evidence that VLMs are highly vulnerable to physically deployable semantic attacks, exposing a previously overlooked robustness gap and underscoring the urgent need for physical-world robustness evaluation of VLMs.