Phantasia: Context-Adaptive Backdoors in Vision Language Models

TL;DR

This paper reveals the vulnerability of vision-language models to easily detectable backdoors and introduces Phantasia, a novel context-adaptive attack that enhances stealth and effectiveness against defenses.

Contribution

It demonstrates the overestimated stealth of existing backdoor attacks and proposes Phantasia, the first context-adaptive backdoor method for VLMs that aligns poisoned outputs with input semantics.

Findings

Existing backdoor attacks are more detectable than previously thought.

Phantasia achieves high attack success rates while remaining stealthy.

Phantasia outperforms prior methods under various defenses.

Abstract

Recent advances in Vision-Language Models (VLMs) have greatly enhanced the integration of visual perception and linguistic reasoning, driving rapid progress in multimodal understanding. Despite these achievements, the security of VLMs, particularly their vulnerability to backdoor attacks, remains significantly underexplored. Existing backdoor attacks on VLMs are still in an early stage of development, with most current methods relying on generating poisoned responses that contain fixed, easily identifiable patterns. In this work, we make two key contributions. First, we demonstrate for the first time that the stealthiness of existing VLM backdoor attacks has been substantially overestimated. By adapting defense techniques originally designed for other domains (e.g., vision-only and text-only models), we show that several state-of-the-art attacks can be detected with surprising ease. Second, to address this gap, we introduce Phantasia, a context-adaptive backdoor attack that dynamically aligns its poisoned outputs with the semantics of each input. Instead of producing static poisoned patterns, Phantasia encourages models to generate contextually coherent yet malicious responses that remain plausible, thereby significantly improving stealth and adaptability. Extensive experiments across diverse VLM architectures reveal that Phantasia achieves state-of-the-art attack success rates while maintaining benign performance under various defensive settings.