Doubly-Universal Adversarial Perturbations: Deceiving Vision-Language Models Across Both Images and Text with a Single Perturbation

TL;DR

This paper introduces Doubly-UAP, a universal adversarial perturbation that can deceive vision-language models across both images and text inputs, revealing vulnerabilities in their attention mechanisms.

Contribution

The paper presents the first doubly-universal adversarial perturbation for VLMs, targeting both image and text modalities with a single optimized attack.

Findings

Doubly-UAP achieves high success rates across multiple vision-language tasks.

Targeting value vectors in attention layers enhances attack effectiveness.

Doubly-UAP outperforms baseline methods in robustness and transferability.

Abstract

Large Vision-Language Models (VLMs) have demonstrated remarkable performance across multimodal tasks by integrating vision encoders with large language models (LLMs). However, these models remain vulnerable to adversarial attacks. Among such attacks, Universal Adversarial Perturbations (UAPs) are especially powerful, as a single optimized perturbation can mislead the model across various input images. In this work, we introduce a novel UAP specifically designed for VLMs: the Doubly-Universal Adversarial Perturbation (Doubly-UAP), capable of universally deceiving VLMs across both image and text inputs. To successfully disrupt the vision encoder's fundamental process, we analyze the core components of the attention mechanism. After identifying value vectors in the middle-to-late layers as the most vulnerable, we optimize Doubly-UAP in a label-free manner with a frozen model. Despite being developed as a black-box to the LLM, Doubly-UAP achieves high attack success rates on VLMs, consistently outperforming baseline methods across vision-language tasks. Extensive ablation studies and analyses further demonstrate the robustness of Doubly-UAP and provide insights into how it influences internal attention mechanisms.