Arbiter: Detecting Interference in LLM Agent System Prompts

TL;DR

This paper introduces Arbiter, a framework that uses formal rules and multi-model analysis to detect interference issues in system prompts for LLM-based coding agents, revealing vulnerabilities and structural problems across major vendors.

Contribution

Arbiter is the first framework combining formal evaluation with multi-model scouring to identify interference patterns in LLM system prompts, improving prompt robustness analysis.

Findings

Identified 152 issues across three major coding agent prompts.

Discovered that prompt architecture influences failure types.

Multi-model analysis reveals different vulnerabilities than single-model methods.

Abstract

System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: \$0.27 USD.