V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs

TL;DR

V-Attack introduces a novel approach to adversarial attacks on LVLMs by targeting disentangled value features, enabling precise manipulation of image semantics and exposing vulnerabilities in current models.

Contribution

The paper proposes V-Attack, a new method that leverages value features within transformer attention to improve controllability and effectiveness of semantic adversarial attacks on LVLMs.

Findings

V-Attack increases attack success rate by 36% on average.

Targeting value features improves semantic manipulation precision.

V-Attack exposes vulnerabilities in multiple LVLMs.

Abstract

Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.