When and Where to Attack? Stage-wise Attention-Guided Adversarial Attack on Large Vision Language Models

TL;DR

This paper introduces SAGA, a stage-wise attention-guided adversarial attack method that efficiently targets high-attention regions in large vision-language models, achieving high success rates with minimal perceptibility.

Contribution

The paper proposes a novel attention-guided attack framework that progressively focuses perturbations on salient regions, improving attack efficiency and success over existing methods.

Findings

SAGA outperforms existing attacks in success rate across ten LVLMs.

Attacking high-attention regions causes structured attention redistribution.

SAGA produces highly imperceptible adversarial examples.

Abstract

Adversarial attacks against Large Vision-Language Models (LVLMs) are crucial for exposing safety vulnerabilities in modern multimodal systems. Recent attacks based on input transformations, such as random cropping, suggest that spatially localized perturbations can be more effective than global image manipulation. However, randomly cropping the entire image is inherently stochastic and fails to use the limited per-pixel perturbation budget efficiently. We make two key observations: (i) regional attention scores are positively correlated with adversarial loss sensitivity, and (ii) attacking high-attention regions induces a structured redistribution of attention toward subsequent salient regions. Based on these findings, we propose Stage-wise Attention-Guided Attack (SAGA), an attention-guided framework that progressively concentrates perturbations on high-attention regions. SAGA enables more efficient use of constrained perturbation budgets, producing highly imperceptible adversarial examples while consistently achieving state-of-the-art attack success rates across ten LVLMs. The source code is available at https://github.com/jackwaky/SAGA.