Sampling-aware Adversarial Attacks Against Large Language Models

TL;DR

This paper demonstrates that incorporating sampling into adversarial attack strategies on large language models significantly improves attack success and efficiency, highlighting the importance of stochasticity in robustness evaluation.

Contribution

It introduces a sampling-aware attack framework, empirically optimizes the attack process, and proposes a novel entropy-based objective for better robustness assessment.

Findings

Sampling enhances attack success rates by up to 37%

Sampling-based methods improve efficiency by up to two orders of magnitude

Output harmfulness distributions are minimally affected by common optimization strategies

Abstract

To guarantee safe and robust deployment of large language models (LLMs) at scale, it is critical to accurately assess their adversarial robustness. Existing adversarial attacks typically target harmful responses in single-point greedy generations, overlooking the inherently stochastic nature of LLMs and overestimating robustness. We show that for the goal of eliciting harmful responses, repeated sampling of model outputs during the attack complements prompt optimization and serves as a strong and efficient attack vector. By casting attacks as a resource allocation problem between optimization and sampling, we empirically determine compute-optimal trade-offs and show that integrating sampling into existing attacks boosts success rates by up to 37\% and improves efficiency by up to two orders of magnitude. We further analyze how distributions of output harmfulness evolve during an adversarial attack, discovering that many common optimization strategies have little effect on output harmfulness. Finally, we introduce a label-free proof-of-concept objective based on entropy maximization, demonstrating how our sampling-aware perspective enables new optimization targets. Overall, our findings establish the importance of sampling in attacks to accurately assess and strengthen LLM safety at scale.