DiffCAP: Diffusion-based Cumulative Adversarial Purification for Vision Language Models

TL;DR

DiffCAP is a diffusion-based method that effectively defends vision language models against adversarial perturbations by iteratively denoising corrupted images, improving robustness and reliability in multimodal understanding tasks.

Contribution

This paper introduces DiffCAP, a novel diffusion-based adversarial purification technique that enhances VLM robustness with reduced complexity and faster denoising compared to existing methods.

Findings

Outperforms existing defenses across six datasets and three VLMs.

Reduces hyperparameter tuning and diffusion time, speeding up the process.

Provides strong empirical and theoretical support for robustness improvements.

Abstract

Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to perturbations poses a significant threat to their reliability in real-world applications. Despite often being imperceptible to humans, these perturbations can drastically alter model outputs, leading to erroneous interpretations and decisions. This paper introduces DiffCAP, a novel diffusion-based purification strategy that can effectively neutralize adversarial corruptions in VLMs. We observe that adding minimal noise to an adversarially corrupted image significantly alters its latent embedding with respect to VLMs. Building on this insight, DiffCAP cumulatively injects random Gaussian noise into adversarially perturbed input data. This process continues until the embeddings of two consecutive noisy images reach a predefined similarity threshold, indicating a potential approach to neutralize the adversarial effect. Subsequently, a pretrained diffusion model is employed to denoise the stabilized image, recovering a clean representation suitable for the VLMs to produce an output. Through extensive experiments across six datasets with three VLMs under varying attack strengths in three task scenarios, we show that DiffCAP consistently outperforms existing defense techniques by a substantial margin. Notably, DiffCAP significantly reduces both hyperparameter tuning complexity and the required diffusion time, thereby accelerating the denoising process. Equipped with strong theoretical and empirical support, DiffCAP provides a robust and practical solution for securely deploying VLMs in adversarial environments.