An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments

TL;DR

This study investigates privacy leakage via prompt injection attacks in black-box chatbots, demonstrating how attackers can hijack tasks and exfiltrate data through crafted external content.

Contribution

It introduces a novel prompt-injection technique called exemplification and evaluates its effectiveness in privacy-leakage scenarios in black-box chatbot environments.

Findings

Prompt injection can hijack chatbot tasks effectively.

Exemplification technique outperforms prior fake-completion methods.

Proof-of-concept shows feasible data exfiltration in controlled settings.

Abstract

LLM-based chatbot agents increasingly process user requests by combining natural-language reasoning with external tools such as web browsing. These capabilities improve usability, but they also create attack surfaces when untrusted external content is processed as part of a user' s task. This paper studies a privacy-leakage attack chain based on indirect prompt injection in black-box chatbot environments, where the attacker has no access to model weights, system prompts, or agent implementation details including how a trajectory is actually managed during its processing for a query. We first analyze how an attacker can hijack an agent' s intended task by crafting external content that appears benign to the victim while inducing the agent to execute an attacker-defined objective. We then evaluate a new prompt-injection technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page as few-shot examples before appending the attacker' s objective. We compare its attack success rate with a prior fake-completion technique. Finally, we demonstrate a proof-of-concept data-exfiltration chain using fictitious personal information in a controlled setting. Our results suggest that prompt injection, jailbreak-style instruction steering, and web-tool invocation can be combined into a feasible privacy-leakage path in deployed chatbot agents.