MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

TL;DR

MELON is a novel defense mechanism against indirect prompt injection attacks in AI agents, using re-execution with masked prompts to detect malicious behavior while maintaining utility.

Contribution

We introduce MELON, a new provable defense against IPI attacks that detects malicious actions by comparing original and masked execution trajectories, outperforming existing methods.

Findings

MELON outperforms state-of-the-art defenses in attack prevention.

MELON maintains higher utility compared to existing defenses.

Combining MELON with prompt augmentation further enhances security.

Abstract

Recent research has explored that LLM agents are vulnerable to indirect prompt injection (IPI) attacks, where malicious tasks embedded in tool-retrieved information can redirect the agent to take unauthorized actions. Existing defenses against IPI have significant limitations: either require essential model training resources, lack effectiveness against sophisticated attacks, or harm the normal utilities. We present MELON (Masked re-Execution and TooL comparisON), a novel IPI defense. Our approach builds on the observation that under a successful attack, the agent's next action becomes less dependent on user tasks and more on malicious tasks. Following this, we design MELON to detect attacks by re-executing the agent's trajectory with a masked user prompt modified through a masking function. We identify an attack if the actions generated in the original and masked executions are similar. We also include three key designs to reduce the potential false positives and false negatives. Extensive evaluation on the IPI benchmark AgentDojo demonstrates that MELON outperforms SOTA defenses in both attack prevention and utility preservation. Moreover, we show that combining MELON with a SOTA prompt augmentation defense (denoted as MELON-Aug) further improves its performance. We also conduct a detailed ablation study to validate our key designs. Code is available at https://github.com/kaijiezhu11/MELON.