AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models

TL;DR

AnyAttack introduces a self-supervised, large-scale adversarial attack framework that can target any output in vision-language models without specific labels, exposing systemic vulnerabilities across multiple systems.

Contribution

It presents a novel foundation model approach trained on unlabeled data, enabling flexible, scalable adversarial attacks on diverse vision-language models and commercial systems.

Findings

Effective across five open-source VLMs

Transfers successfully to commercial systems

Reveals systemic vulnerabilities in VLMs

Abstract

Due to their multimodal capabilities, Vision-Language Models (VLMs) have found numerous impactful applications in real-world scenarios. However, recent studies have revealed that VLMs are vulnerable to image-based adversarial attacks. Traditional targeted adversarial attacks require specific targets and labels, limiting their real-world impact.We present AnyAttack, a self-supervised framework that transcends the limitations of conventional attacks through a novel foundation model approach. By pre-training on the massive LAION-400M dataset without label supervision, AnyAttack achieves unprecedented flexibility - enabling any image to be transformed into an attack vector targeting any desired output across different VLMs.This approach fundamentally changes the threat landscape, making adversarial capabilities accessible at an unprecedented scale. Our extensive validation across five open-source VLMs (CLIP, BLIP, BLIP2, InstructBLIP, and MiniGPT-4) demonstrates AnyAttack's effectiveness across diverse multimodal tasks. Most concerning, AnyAttack seamlessly transfers to commercial systems including Google Gemini, Claude Sonnet, Microsoft Copilot and OpenAI GPT, revealing a systemic vulnerability requiring immediate attention.