Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Models

TL;DR

This paper introduces PromptAnatomy, a framework that dissects prompt structures to generate interpretable adversarial examples, revealing the heterogeneous vulnerabilities of prompt components in large language models and improving robustness evaluation.

Contribution

The paper presents PromptAnatomy and ComPerturb, novel methods for dissecting prompts and generating targeted adversarial attacks, addressing the structural heterogeneity overlooked by prior approaches.

Findings

PromptAnatomy effectively dissects prompts into functional components.

ComPerturb achieves state-of-the-art attack success rates across datasets and models.

Prompt structure awareness enhances adversarial robustness evaluation.

Abstract

Prompt-based adversarial attacks have become an effective means to assess the robustness of large language models (LLMs). However, existing approaches often treat prompts as monolithic text, overlooking their structural heterogeneity-different prompt components contribute unequally to adversarial robustness. Prior works like PromptRobust assume prompts are value-neutral, but our analysis reveals that complex, domain-specific prompts with rich structures have components with differing vulnerabilities. To address this gap, we introduce PromptAnatomy, an automated framework that dissects prompts into functional components and generates diverse, interpretable adversarial examples by selectively perturbing each component using our proposed method, ComPerturb. To ensure linguistic plausibility and mitigate distribution shifts, we further incorporate a perplexity (PPL)-based filtering mechanism. As a complementary resource, we annotate four public instruction-tuning datasets using the PromptAnatomy framework, verified through human review. Extensive experiments across these datasets and five advanced LLMs demonstrate that ComPerturb achieves state-of-the-art attack success rates. Ablation studies validate the complementary benefits of prompt dissection and PPL filtering. Our results underscore the importance of prompt structure awareness and controlled perturbation for reliable adversarial robustness evaluation in LLMs. Code and data are available at https://github.com/Yujiaaaaa/PACP.