May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks

TL;DR

This paper demonstrates that fine-tuning based prompt injection defenses for large language models are vulnerable to new architecture-aware attacks, achieving high success rates and challenging their security claims.

Contribution

It introduces a novel attention-based attack algorithm that breaks recent whitebox defenses, revealing their vulnerabilities in the context of prompt injection.

Findings

Attacks achieve 85-95% success rate on unseen prompts

Fine-tuning defenses do not provide robust security in whitebox settings

The new attack method is effective with modest attacker resources

Abstract

A popular class of defenses against prompt injection attacks on large language models (LLMs) relies on fine-tuning to separate instructions and data, so that the LLM does not follow instructions that might be present with data. We evaluate the robustness of this approach in the whitebox setting by constructing strong optimization-based attacks, and show that the defenses do not provide the claimed security properties. Specifically, we construct a novel attention-based attack algorithm for textual LLMs and apply it to three recent whitebox defenses SecAlign (CCS 2025), SecAlign++, and StruQ (USENIX Security 2025), showing attacks with success rates of up to \textbf{85-95\%} on unseen prompts with modest increase in attacker budget in terms of tokens. Our findings make fundamental progress towards understanding the robustness of prompt injection defenses in the whitebox setting. We release our code and attacks at https://github.com/nishitvp/better_opts_attacks