Toward a Dynamic Stackelberg Game-Theoretic Framework for Agentic AI Defense Against LLM Jailbreaking

TL;DR

This paper introduces a game-theoretic framework combining extensive form games and RRT search to model and analyze interactions between prompt engineers and LLMs, aiming to improve AI safety against jailbreaks.

Contribution

It presents a novel dynamic Stackelberg game model with RRT exploration for understanding and defending against LLM jailbreak strategies, providing a theoretical basis for AI safety.

Findings

The framework captures both jailbreak discovery and strategic responses.

The local Stackelberg equilibrium explains when attacks become unprofitable.

Provides a foundation for evaluating and strengthening LLM guardrails.

Abstract

This paper proposes a game theoretic framework that models the interaction between prompt engineers and large language models (LLMs) as a two player extensive form game coupled with a Rapidly exploring Random Trees (RRT) search over prompt space. The attacker incrementally samples, extends, and tests prompts, while the LLM chooses to accept, reject, or redirect, leading to terminal outcomes of Safe Interaction, Blocked, or Jailbreak. Embedding RRT exploration inside the extensive form game captures both the discovery phase of jailbreak strategies and the strategic responses of the model. Furthermore, we show that the defender behavior can be interpreted through a local Stackelberg equilibrium condition, which explains when the attacker can no longer obtain profitable prompt deviations and provides a theoretical lens for understanding the effectiveness of our Purple Agent defense. The resulting game tree thus offers a principled foundation for evaluating, interpreting, and hardening LLM guardrails.