Doppelganger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack

TL;DR

This paper introduces the Doppelganger method to attack LLM-based agents by exposing internal prompts and information, and proposes the CAT prompt as a defense, highlighting vulnerabilities in prompt-based autonomous systems.

Contribution

The paper presents a novel adversarial attack method on LLM agents and a corresponding prompt-based defense mechanism, revealing security vulnerabilities in prompt engineering.

Findings

Doppelganger method can successfully expose internal prompts.

The attack compromises agent consistency and internal information.

CAT prompts effectively defend against the attack.

Abstract

Since the advent of large language models, prompt engineering now enables the rapid, low-effort creation of diverse autonomous agents that are already in widespread use. Yet this convenience raises urgent concerns about the safety, robustness, and behavioral consistency of the underlying prompts, along with the pressing challenge of preventing those prompts from being exposed to user's attempts. In this paper, we propose the ''Doppelganger method'' to demonstrate the risk of an agent being hijacked, thereby exposing system instructions and internal information. Next, we define the ''Prompt Alignment Collapse under Adversarial Transfer (PACAT)'' level to evaluate the vulnerability to this adversarial transfer attack. We also propose a ''Caution for Adversarial Transfer (CAT)'' prompt to counter the Doppelganger method. The experimental results demonstrate that the Doppelganger method can compromise the agent's consistency and expose its internal information. In contrast, CAT prompts enable effective defense against this adversarial attack.