No Query, No Access

TL;DR

This paper introduces VDBA, a novel adversarial attack method that effectively fools NLP models, including large language models, without requiring model access or extensive queries, highlighting significant security risks.

Contribution

VDBA is a new attack approach that uses victim texts and shadow datasets to generate effective adversarial examples without model access or many queries.

Findings

VDBA achieves a 52.08% increase in attack success rate over state-of-the-art methods.

VDBA reduces attack queries to zero, demonstrating high efficiency.

VDBA poses a serious threat to LLMs like Qwen2 and GPT models, with an ASR of 45.99%.

Abstract

Textual adversarial attacks mislead NLP models, including Large Language Models (LLMs), by subtly modifying text. While effective, existing attacks often require knowledge of the victim model, extensive queries, or access to training data, limiting real-world feasibility. To overcome these constraints, we introduce the \textbf{Victim Data-based Adversarial Attack (VDBA)}, which operates using only victim texts. To prevent access to the victim model, we create a shadow dataset with publicly available pre-trained models and clustering methods as a foundation for developing substitute models. To address the low attack success rate (ASR) due to insufficient information feedback, we propose the hierarchical substitution model design, generating substitute models to mitigate the failure of a single substitute model at the decision boundary. Concurrently, we use diverse adversarial example generation, employing various attack methods to generate and select the adversarial example with better similarity and attack effectiveness. Experiments on the Emotion and SST5 datasets show that VDBA outperforms state-of-the-art methods, achieving an ASR improvement of 52.08\% while significantly reducing attack queries to 0. More importantly, we discover that VDBA poses a significant threat to LLMs such as Qwen2 and the GPT family, and achieves the highest ASR of 45.99% even without access to the API, confirming that advanced NLP models still face serious security risks. Our codes can be found at https://anonymous.4open.science/r/VDBA-Victim-Data-based-Adversarial-Attack-36EC/