IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding

TL;DR

This paper introduces IAG, a novel input-aware backdoor attack on vision-language models for visual grounding, demonstrating high effectiveness and stealth without degrading normal performance, revealing significant security vulnerabilities.

Contribution

We propose IAG, the first dynamic, input-aware backdoor attack on VLM-based visual grounding that uses text-guided triggers conditioned on target descriptions.

Findings

IAG achieves high attack success rates across multiple models and datasets.

IAG maintains normal grounding accuracy on benign inputs.

The attack is robust against existing defenses and transferable across models.

Abstract

Recent advances in vision-language models (VLMs) have significantly enhanced the visual grounding task, which involves locating objects in an image based on natural language queries. Despite these advancements, the security of VLM-based grounding systems has not been thoroughly investigated. This paper reveals a novel and realistic vulnerability: the first multi-target backdoor attack on VLM-based visual grounding. Unlike prior attacks that rely on static triggers or fixed targets, we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack. This is achieved through a text-conditioned UNet that embeds imperceptible target semantic cues into visual inputs while preserving normal grounding performance on benign samples. We further develop a joint training objective that balances language capability with perceptual reconstruction to ensure imperceptibility, effectiveness, and stealth. Extensive experiments on multiple VLMs (e.g., LLaVA, InternVL, Ferret) and benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Flickr30k Entities, and ShowUI) demonstrate that IAG achieves the best ASRs compared with other baselines on almost all settings without compromising clean accuracy, maintaining robustness against existing defenses, and exhibiting transferability across datasets and models. These findings underscore critical security risks in grounding-capable VLMs and highlight the need for further research on trustworthy multimodal understanding.