A Theoretical Game of Attacks via Compositional Skills

TL;DR

This paper develops a theoretical framework modeling adversarial attacks and defenses on large language models, revealing optimal strategies and demonstrating improved attack performance empirically.

Contribution

It introduces a formal game-theoretic model for adversarial prompts, derives optimal attack and defense strategies, and empirically validates the attack's effectiveness.

Findings

Theoretical best-response attack closely relates to existing methods.

Optimal defense strategy can be derived with provable guarantees.

Empirical evaluation shows stronger attack performance across models and benchmarks.

Abstract

As large language models grow increasingly capable, concerns about their safe deployment have intensified. While numerous alignment strategies aim to restrict harmful behavior, these defenses can still be circumvented through carefully designed adversarial prompts. In this work, we introduce a theoretical framework that formalizes a game between an attacker and a defender. Within this framework, we design a theoretical best-response attack strategy and show that it is closely related to many existing adversarial prompting methods. We further analyze the resulting game, characterize its equilibria, and reveal inherent advantages for the attacker. Drawing on our theoretical analysis, we also derive a provably optimal defense strategy. Empirically, we evaluate a practical instantiation of the theoretically optimal attack and observe stronger performance relative to existing adversarial prompting approaches in diverse settings encompassing different LLMs and benchmarks.