Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries

TL;DR

This paper introduces CEMA, a black-box multi-task adversarial attack method that efficiently generates adversarial texts across various tasks using minimal queries and transferability, even against commercial APIs and large models.

Contribution

CEMA is a novel multi-task attack framework that leverages deep substitute models and transferability, enabling effective attacks with limited queries across diverse tasks and models.

Findings

Achieves high attack success with as few as 100 queries.

Effective against multiple tasks including classification, translation, and image generation.

Successfully targets commercial APIs and large language models.

Abstract

Current multi-task adversarial text attacks rely on abundant access to shared internal features and numerous queries, often limited to a single task type. As a result, these attacks are less effective against practical scenarios involving black-box feedback APIs, limited queries, or multiple task types. To bridge this gap, we propose \textbf{C}luster and \textbf{E}nsemble \textbf{M}ulti-task Text Adversarial \textbf{A}ttack (\textbf{CEMA}), an effective black-box attack that exploits the transferability of adversarial texts across different tasks. CEMA simplifies complex multi-task scenarios by using a \textit{deep-level substitute model} trained in a \textit{plug-and-play} manner for text classification, enabling attacks without mimicking the victim model. This approach requires only a few queries for training, converting multi-task attacks into classification attacks and allowing attacks across various tasks. CEMA generates multiple adversarial candidates using different text classification methods and selects the one that most effectively attacks substitute models. In experiments involving multi-task models with two, three, or six tasks--spanning classification, translation, summarization, and text-to-image generation--CEMA demonstrates significant attack success with as few as 100 queries. Furthermore, CEMA can target commercial APIs (e.g., Baidu and Google Translate), large language models (e.g., ChatGPT 4o), and image-generation models (e.g., Stable Diffusion V2), showcasing its versatility and effectiveness in real-world applications.