The Evaluation Game: Beyond Static LLM Benchmarking

TL;DR

This paper introduces a game-theoretic framework to analyze the robustness of large language models against jailbreaks, emphasizing the importance of data augmentation and local generalization in fine-tuning.

Contribution

It formalizes the interaction between evaluators and trainers as a two-player game using group actions, providing new insights into adversarial robustness and evaluation dynamics.

Findings

Evaluator maintains a constant miss ratio below a critical threshold.

Fine-tuning induces local generalization, with refusal rates correlated to prompt distance.

The framework recasts benchmarks as orbits under group actions, challenging static evaluation methods.

Abstract

As jailbreaks, adversarially crafted inputs that bypass safety constraints, continue to be discovered in Large Language Models, practitioners increasingly rely on fine-tuning as a defensive strategy. Yet the theoretical foundations underlying this robustness fine-tuning remain underexplored. We introduce a game-theoretic framework in which the interaction between an evaluator (auditing the model for jailbreaks) and a trainer is formalized as a two-player game. A key feature of our approach is the use of group actions, a mathematical structure that captures symmetries and transformations, to formally represent data augmentation. The simplest non-trivial instance is the circle with cyclic translation groups, where we exhibit various regimes depending on the trainer's generalization range. Below a critical threshold, the evaluator maintains a constant miss ratio for linearly many rounds, whereas other settings can yield very different behaviors. We further provide empirical evidence supporting locality-dependence of the model: for the three model families we tested (Llama, Qwen and Mistral), we have significant evidence that fine-tuning on adversarial prompts induces only local generalization, with refusal rates on test examples highly correlated with the distance to the fine-tuning prompts. Our framework recasts the central object of adversarial evaluation: a benchmark is not a static set of prompts but an orbit under the evaluator's group action, and audit protocols that ignore trainer-side adaptation cannot distinguish a genuine fix from a memorized patch.