Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution

TL;DR

This paper investigates how prompt injection attacks can cause LLM-based agents to leak personal data during task execution, revealing vulnerabilities in agent security and safety measures.

Contribution

It introduces data flow-based prompt injection attacks in a banking scenario and evaluates their impact on agent utility and data leakage across multiple tasks.

Findings

LLMs experience a 15-50% utility drop under attack

Average attack success rate (ASR) is around 20% in initial tests

Some defenses can reduce ASR to zero

Abstract

Previous benchmarks on prompt injection in large language models (LLMs) have primarily focused on generic tasks and attacks, offering limited insights into more complex threats like data exfiltration. This paper examines how prompt injection can cause tool-calling agents to leak personal data observed during task execution. Using a fictitious banking agent, we develop data flow-based attacks and integrate them into AgentDojo, a recent benchmark for agentic security. To enhance its scope, we also create a richer synthetic dataset of human-AI banking conversations. In 16 user tasks from AgentDojo, LLMs show a 15-50 percentage point drop in utility under attack, with average attack success rates (ASR) around 20 percent; some defenses reduce ASR to zero. Most LLMs, even when successfully tricked by the attack, avoid leaking highly sensitive data like passwords, likely due to safety alignments, but they remain vulnerable to disclosing other personal data. The likelihood of password leakage increases when a password is requested along with one or two additional personal details. In an extended evaluation across 48 tasks, the average ASR is around 15 percent, with no built-in AgentDojo defense fully preventing leakage. Tasks involving data extraction or authorization workflows, which closely resemble the structure of exfiltration attacks, exhibit the highest ASRs, highlighting the interaction between task type, agent performance, and defense efficacy.