STACK: Adversarial Attacks on LLM Safeguard Pipelines

TL;DR

This paper evaluates the security of AI safeguard pipelines, introduces a staged attack method called STACK, and demonstrates its effectiveness in bypassing defenses, highlighting the need for improved safeguards.

Contribution

It develops a novel staged attack method (STACK) against AI safeguard pipelines and evaluates its effectiveness, revealing vulnerabilities in current defense strategies.

Findings

A new few-shot classifier outperforms existing safeguards with 0% attack success rate.

STACK achieves 71% attack success in black-box settings.

Transfer attacks achieve 33% success, indicating transferability of attacks.

Abstract

Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.