Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation

TL;DR

This paper uncovers a novel semantic backdoor attack on vision-language models that manipulates cross-modal semantic alignment, demonstrating high attack success rates and resistance to defenses, highlighting critical security vulnerabilities.

Contribution

The paper introduces BadSem, a new semantic backdoor attack leveraging cross-modal mismatches, and constructs SIMBad dataset for evaluation, revealing vulnerabilities in VLMs.

Findings

Achieves over 98% attack success rate across models

Generalizes well to out-of-distribution datasets

Defenses based on prompts and fine-tuning are ineffective

Abstract

Vision Language Models (VLMs) have shown remarkable performance, but are also vulnerable to backdoor attacks whereby the adversary can manipulate the model's outputs through hidden triggers. Prior attacks primarily rely on single-modality triggers, leaving the crucial cross-modal fusion nature of VLMs largely unexplored. Unlike prior work, we identify a novel attack surface that leverages cross-modal semantic mismatches as implicit triggers. Based on this insight, we propose BadSem (Backdoor Attack with Semantic Manipulation), a data poisoning attack that injects stealthy backdoors by deliberately misaligning image-text pairs during training. To perform the attack, we construct SIMBad, a dataset tailored for semantic manipulation involving color and object attributes. Extensive experiments across four widely used VLMs show that BadSem achieves over 98% average ASR, generalizes well to out-of-distribution datasets, and can transfer across poisoning modalities. Our detailed analysis using attention visualization shows that backdoored models focus on semantically sensitive regions under mismatched conditions while maintaining normal behavior on clean inputs. To mitigate the attack, we try two defense strategies based on system prompt and supervised fine-tuning but find that both of them fail to mitigate the semantic backdoor. Our findings highlight the urgent need to address semantic vulnerabilities in VLMs for their safer deployment.