Searching for Privacy Risks in LLM Agents via Simulation

TL;DR

This paper introduces a simulation-based framework using LLMs to identify privacy risks in multi-turn interactions of LLM agents, revealing evolving attack and defense strategies.

Contribution

It presents a novel search-based method employing LLMs for analyzing and improving privacy attack and defense strategies in agent interactions.

Findings

Attack strategies escalate from requests to impersonation and forgery.

Defense strategies evolve from rule-based constraints to identity-verification.

Strategies generalize across different models and scenarios.

Abstract

The widespread deployment of LLM-based agents is likely to introduce a critical privacy threat: malicious agents that proactively engage others in multi-turn interactions to extract sensitive information. However, the evolving nature of such dynamic dialogues makes it challenging to anticipate emerging vulnerabilities and design effective defenses. To tackle this problem, we present a search-based framework that alternates between improving attack and defense strategies through the simulation of privacy-critical agent interactions. Specifically, we employ LLMs as optimizers to analyze simulation trajectories and iteratively propose new agent instructions. To explore the strategy space more efficiently, we further utilize parallel search with multiple threads and cross-thread propagation. Through this process, we find that attack strategies escalate from direct requests to sophisticated tactics, such as impersonation and consent forgery, while defenses evolve from simple rule-based constraints to robust identity-verification state machines. The discovered attacks and defenses generalize across diverse scenarios and backbone models, providing useful insights for developing privacy-aware agents.