CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models

TL;DR

This paper introduces CBV, a novel backdoor attack on vision-language models using diffusion models to generate natural, stealthy poisoned samples with high attack success rates, avoiding detection by traditional methods.

Contribution

We propose a diffusion model-based backdoor attack that creates natural poisoned samples guided by multimodal and GradCAM information, improving stealthiness and effectiveness.

Findings

Achieves over 80% attack success rate on multiple VLMs.

Generates natural poisoned samples that are hard to detect.

Maintains normal model functionality despite the attack.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in tasks such as image captioning and visual question answering (VQA). However, as their applications become increasingly widespread, recent studies have revealed that VLMs are vulnerable to backdoor attacks. Existing backdoor attacks on VLMs primarily rely on data poisoning by adding visual triggers and modifying text labels, where the induced image-text mismatch makes poisoned samples easy to detect. To address this limitation, we propose the Clean-Label Backdoor Attack on VLMs via Diffusion Models (CBV), which leverages diffusion models to generate natural poisoned examples via score matching. Specifically, CBV modifies the score during the reverse generation process of the diffusion model to guide the generation of poisoned samples that contain triggered image features. To further enhance the effectiveness of the attack, we incorporate the textual information of the triggered images as multimodal guidance during generation. Moreover, to enhance stealthiness, we introduce a GradCAM-guided Mask (GM) that restricts modifications to only the most semantically important regions, rather than the entire image. We evaluate our method on MSCOCO and VQA v2 with four representative VLMs, achieving over 80% ASR while preserving normal functionality.