Concealment of Intent: A Game-Theoretic Analysis

TL;DR

This paper introduces a game-theoretic framework to analyze intent-hiding adversarial prompts in large language models, revealing vulnerabilities and proposing defenses to improve safety against malicious manipulations.

Contribution

It presents a novel game-theoretic model for intent-hiding attacks and defenses in LLMs, providing insights into attack structures and countermeasures.

Findings

Intent-hiding prompts effectively conceal malicious intent.

Attacker has structural advantages in the game-theoretic model.

Proposed defenses reduce attack success across multiple LLMs.

Abstract

As large language models (LLMs) grow more capable, concerns about their safe deployment have also grown. Although alignment mechanisms have been introduced to deter misuse, they remain vulnerable to carefully designed adversarial prompts. In this work, we present a scalable attack strategy: intent-hiding adversarial prompting, which conceals malicious intent through the composition of skills. We develop a game-theoretic framework to model the interaction between such attacks and defense systems that apply both prompt and response filtering. Our analysis identifies equilibrium points and reveals structural advantages for the attacker. To counter these threats, we propose and analyze a defense mechanism tailored to intent-hiding attacks. Empirically, we validate the attack's effectiveness on multiple real-world LLMs across a range of malicious behaviors, demonstrating clear advantages over existing adversarial prompting techniques.