Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations

TL;DR

This paper uncovers how imperceptible visual cues can deceive multimodal agents in financial tasks, proposing a novel attack method and defenses to improve robustness.

Contribution

It introduces PriceBlind, a white-box adversarial attack exploiting modality gaps in CLIP-based encoders for screenshot-based evaluation.

Findings

PriceBlind achieves 80% ASR in white-box evaluation.

Transfer attacks reach 35-41% ASR across multiple models.

Robust encoders and defenses reduce ASR but affect accuracy.

Abstract

The rapid proliferation of Multimodal Large Language Models (MLLMs) has enabled mobile agents to execute high-stakes financial transactions, but their adversarial robustness remains underexplored. We identify Visual Dominance Hallucination (VDH), where imperceptible visual cues can override textual price evidence in screenshot-based, price-constrained settings and lead agents to irrational decisions. We propose PriceBlind, a stealthy white-box adversarial attack framework for controlled screenshot-based evaluation. PriceBlind exploits the modality gap in CLIP-based encoders via a Semantic-Decoupling Loss that aligns the image embedding with low-cost, value-associated anchors while preserving pixel-level fidelity. On E-ShopBench, PriceBlind achieves around 80% ASR in white-box evaluation; under a simplified single-turn coordinate-selection protocol, Ensemble-DI-FGSM transfers with roughly 35-41% ASR across GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet. We also show that robust encoders and Verify-then-Act defenses reduce ASR substantially, though with some clean-accuracy trade-off.