One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP

TL;DR

This paper introduces UnivIntruder, a novel adversarial attack method that uses a single CLIP model and textual concepts to generate universal, transferable, and targeted adversarial examples, exposing vulnerabilities in various AI systems without needing target model queries.

Contribution

The paper presents UnivIntruder, a new attack framework that leverages CLIP and textual concepts to craft effective adversarial examples without access to target models or training data.

Findings

Achieves up to 85% attack success rate on ImageNet

Successfully compromises image search engines and language models

Outperforms existing transfer-based attack methods

Abstract

Deep Neural Networks (DNNs) have achieved widespread success yet remain prone to adversarial attacks. Typically, such attacks either involve frequent queries to the target model or rely on surrogate models closely mirroring the target model -- often trained with subsets of the target model's training data -- to achieve high attack success rates through transferability. However, in realistic scenarios where training data is inaccessible and excessive queries can raise alarms, crafting adversarial examples becomes more challenging. In this paper, we present UnivIntruder, a novel attack framework that relies solely on a single, publicly available CLIP model and publicly available datasets. By using textual concepts, UnivIntruder generates universal, transferable, and targeted adversarial perturbations that mislead DNNs into misclassifying inputs into adversary-specified classes defined by textual concepts. Our extensive experiments show that our approach achieves an Attack Success Rate (ASR) of up to 85% on ImageNet and over 99% on CIFAR-10, significantly outperforming existing transfer-based methods. Additionally, we reveal real-world vulnerabilities, showing that even without querying target models, UnivIntruder compromises image search engines like Google and Baidu with ASR rates up to 84%, and vision language models like GPT-4 and Claude-3.5 with ASR rates up to 80%. These findings underscore the practicality of our attack in scenarios where traditional avenues are blocked, highlighting the need to reevaluate security paradigms in AI applications.