To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt

TL;DR

This paper introduces Polymorphic Prompt Assembling, a lightweight method that dynamically varies prompt structures to defend LLM agents against prompt injection attacks without significant performance loss.

Contribution

The paper presents a novel, dynamic prompt variation technique that effectively prevents prompt injection attacks with minimal overhead.

Findings

PPA significantly reduces success rate of prompt injection attacks.

PPA maintains high performance and usability of LLM agents.

Compared to existing defenses, PPA offers superior security with lower overhead.

Abstract

LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model's behavior. Traditional defenses like input sanitization, guard models, and guardrails are either cumbersome or ineffective. In this paper, we propose a novel, lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which protects against prompt injection with near-zero overhead. The approach is based on the insight that prompt injection requires guessing and breaking the structure of the system prompt. By dynamically varying the structure of system prompts, PPA prevents attackers from predicting the prompt structure, thereby enhancing security without compromising performance. We conducted experiments to evaluate the effectiveness of PPA against existing attacks and compared it with other defense methods.