Defense Against Indirect Prompt Injection via Tool Result Parsing

TL;DR

This paper introduces a novel defense method against indirect prompt injection in LLM agents, using tool result parsing to filter malicious code, achieving low attack success rates and high utility.

Contribution

The paper presents a new approach that enhances robustness against IPI by parsing tool results, outperforming existing detection and prompt-based defenses.

Findings

Achieves the lowest Attack Success Rate (ASR) among compared methods.

Maintains high Utility under Attack (UA) with the proposed approach.

Outperforms existing defenses in robustness against sophisticated injection attacks.

Abstract

As LLM agents transition from digital assistants to physical controllers in autonomous systems and robotics, they face an escalating threat from indirect prompt injection. By embedding adversarial instructions into the results of tool calls, attackers can hijack the agent's decision-making process to execute unauthorized actions. This vulnerability poses a significant risk as agents gain more direct control over physical environments. Existing defense mechanisms against Indirect Prompt Injection (IPI) generally fall into two categories. The first involves training dedicated detection models; however, this approach entails high computational overhead for both training and inference, and requires frequent updates to keep pace with evolving attack vectors. Alternatively, prompt-based methods leverage the inherent capabilities of LLMs to detect or ignore malicious instructions via prompt engineering. Despite their flexibility, most current prompt-based defenses suffer from high Attack Success Rates (ASR), demonstrating limited robustness against sophisticated injection attacks. In this paper, we propose a novel method that provides LLMs with precise data via tool result parsing while effectively filtering out injected malicious code. Our approach achieves competitive Utility under Attack (UA) while maintaining the lowest Attack Success Rate (ASR) to date, significantly outperforming existing methods. Code is available at GitHub.