A Critical Evaluation of Defenses against Prompt Injection Attacks

TL;DR

This paper critically evaluates existing defenses against prompt injection attacks on LLMs, revealing that many are less effective than previously claimed when assessed with a comprehensive, principled methodology.

Contribution

It introduces a rigorous evaluation framework for defenses against prompt injection, highlighting gaps in prior assessments and guiding future defense development.

Findings

Existing defenses are less effective than previously claimed.

Many defenses compromise LLM utility under evaluation.

The paper proposes a comprehensive evaluation methodology.

Abstract

Large Language Models (LLMs) are vulnerable to prompt injection attacks, and several defenses have recently been proposed, often claiming to mitigate these attacks successfully. However, we argue that existing studies lack a principled approach to evaluating these defenses. In this paper, we argue the need to assess defenses across two critical dimensions: (1) effectiveness, measured against both existing and adaptive prompt injection attacks involving diverse target and injected prompts, and (2) general-purpose utility, ensuring that the defense does not compromise the foundational capabilities of the LLM. Our critical evaluation reveals that prior studies have not followed such a comprehensive evaluation methodology. When assessed using this principled approach, we show that existing defenses are not as successful as previously reported. This work provides a foundation for evaluating future defenses and guiding their development. Our code and data are available at: https://github.com/PIEval123/PIEval.