Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents

TL;DR

This paper introduces VIBMA, a novel clean-text backdoor attack on VLM-based mobile agents that manipulates visual inputs to embed stealthy malicious behaviors without altering textual prompts, exposing security vulnerabilities.

Contribution

It presents the first attack targeting VLM-based mobile agents using visual triggers, demonstrating high success rates and stealthiness, and highlights security risks in mobile agent fine-tuning.

Findings

High attack success rate (up to 94.67%)

Maintains task performance with minimal impact (up to 95.85% FSR)

Effective triggers include static, dynamic, and blended patterns

Abstract

Mobile agents powered by vision-language models (VLMs) are increasingly adopted for tasks such as UI automation and camera-based assistance. These agents are typically fine-tuned using small-scale, user-collected data, making them susceptible to stealthy training-time threats. This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents. The attack injects malicious behaviors into the model by modifying only the visual input while preserving textual prompts and instructions, achieving stealth through the complete absence of textual anomalies. Once the agent is fine-tuned on this poisoned data, adding a predefined visual pattern (trigger) at inference time activates the attacker-specified behavior (backdoor). Our attack aligns the training gradients of poisoned samples with those of an attacker-specified target instance, effectively embedding backdoor-specific features into the poisoned data. To ensure the robustness and stealthiness of the attack, we design three trigger variants that better resemble real-world scenarios: static patches, dynamic motion patterns, and low-opacity blended content. Extensive experiments on six Android applications and three mobile-compatible VLMs demonstrate that our attack achieves high success rates (ASR up to 94.67%) while preserving clean-task behavior (FSR up to 95.85%). We further conduct ablation studies to understand how key design factors impact attack reliability and stealth. These findings is the first to reveal the security vulnerabilities of mobile agents and their susceptibility to backdoor injection, underscoring the need for robust defenses in mobile agent adaptation pipelines.