CAPTURE: Context-Aware Prompt Injection Testing and Robustness Enhancement

TL;DR

CAPTURE introduces a new context-aware benchmark for prompt injection, revealing current guardrails' limitations and demonstrating a trained model that significantly improves detection accuracy and robustness.

Contribution

The paper presents CAPTURE, a novel benchmark for context-aware prompt injection testing, and introduces CaptureGuard, a model that enhances detection and reduces false positives and negatives.

Findings

Current guardrails have high false negatives in adversarial cases.

Existing models suffer from over-defense, causing false positives.

CaptureGuard outperforms existing defenses on context-aware datasets.

Abstract

Prompt injection remains a major security risk for large language models. However, the efficacy of existing guardrail models in context-aware settings remains underexplored, as they often rely on static attack benchmarks. Additionally, they have over-defense tendencies. We introduce CAPTURE, a novel context-aware benchmark assessing both attack detection and over-defense tendencies with minimal in-domain examples. Our experiments reveal that current prompt injection guardrail models suffer from high false negatives in adversarial cases and excessive false positives in benign scenarios, highlighting critical limitations. To demonstrate our framework's utility, we train CaptureGuard on our generated data. This new model drastically reduces both false negative and false positive rates on our context-aware datasets while also generalizing effectively to external benchmarks, establishing a path toward more robust and practical prompt injection defenses.