PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention

TL;DR

PA-Attack introduces a novel gray-box attack method on LVLMs that uses prototypes and attention mechanisms to improve attack transferability, efficiency, and task generalization across diverse models and tasks.

Contribution

It proposes a prototype-anchored guidance and a two-stage attention mechanism to enhance adversarial attacks on LVLMs, addressing limitations of prior white-box and black-box methods.

Findings

Achieves an average 75.1% score reduction rate across tasks.

Demonstrates strong attack effectiveness and efficiency.

Shows improved task generalization across LVLM architectures.

Abstract

Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box attacks rarely generalize across tasks, and black-box methods depend on expensive transfer, which limits efficiency. The vision encoder, standardized and often shared across LVLMs, provides a stable gray-box pivot with strong cross-model transfer. Building on this premise, we introduce PA-Attack (Prototype-Anchored Attentive Attack). PA-Attack begins with a prototype-anchored guidance that provides a stable attack direction towards a general and dissimilar prototype, tackling the attribute-restricted issue and limited task generalization of vanilla attacks. Building on this, we propose a two-stage attention enhancement mechanism: (i) leverage token-level attention scores to concentrate perturbations on critical visual tokens, and (ii) adaptively recalibrate attention weights to track the evolving attention during the adversarial process. Extensive experiments across diverse downstream tasks and LVLM architectures show that PA-Attack achieves an average 75.1% score reduction rate (SRR), demonstrating strong attack effectiveness, efficiency, and task generalization in LVLMs. Code is available at https://github.com/hefeimei06/PA-Attack.