InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

TL;DR

InjecAgent is a comprehensive benchmark that evaluates the vulnerability of tool-integrated large language model agents to indirect prompt injection attacks, revealing significant security risks and the need for mitigation strategies.

Contribution

This work introduces InjecAgent, the first benchmark specifically designed to assess IPI attack vulnerabilities in LLM agents across multiple tools and attacker scenarios.

Findings

Agents are vulnerable to IPI attacks, with GPT-4 being compromised 24% of the time.

Reinforcing attacker instructions with hacking prompts increases attack success rates.

The benchmark covers 1,054 test cases across 17 user tools and 62 attacker tools.

Abstract

Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative. In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4. Our findings raise questions about the widespread deployment of LLM Agents. Our benchmark is available at https://github.com/uiuc-kang-lab/InjecAgent.