LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents

TL;DR

LaSM is a layer-wise attention scaling method that enhances GUI agent robustness against pop-up attacks without retraining, by aligning attention with task-relevant regions.

Contribution

It uncovers layer-wise attention divergence patterns and introduces LaSM, a novel attention amplification technique that improves defense success without additional training.

Findings

LaSM significantly increases attack defense success rate.

It maintains the model's general capabilities with negligible impact.

Attention misalignment is identified as a core vulnerability.

Abstract

Graphical user interface (GUI) agents built on multimodal large language models (MLLMs) have recently demonstrated strong decision-making abilities in screen-based interaction tasks. However, they remain highly vulnerable to pop-up-based environmental injection attacks, where malicious visual elements divert model attention and lead to unsafe or incorrect actions. Existing defense methods either require costly retraining or perform poorly under inductive interference. In this work, we systematically study how such attacks alter the attention behavior of GUI agents and uncover a layer-wise attention divergence pattern between correct and incorrect outputs. Based on this insight, we propose \textbf{LaSM}, a \textit{Layer-wise Scaling Mechanism} that selectively amplifies attention and MLP modules in critical layers. LaSM improves the alignment between model saliency and task-relevant regions without additional training. Extensive experiments across multiple datasets demonstrate that our method significantly improves the defense success rate and exhibits strong robustness, while having negligible impact on the model's general capabilities. Our findings reveal that attention misalignment is a core vulnerability in MLLM agents and can be effectively addressed through selective layer-wise modulation. Our code can be found in https://github.com/YANGTUOMAO/LaSM.