Judge Circuits

TL;DR

This paper investigates how large language models' judgment consistency is affected by output formatting, revealing a shared internal sub-graph responsible for evaluations and how formatting impacts judgment signals.

Contribution

It introduces PEAP to causally analyze internal mechanisms, identifying a shared latent evaluator sub-graph and decoupling judgment from output format.

Findings

Judgments across tasks share a sparse, generalized sub-graph in MLPs.

Zero-ablating this sub-graph collapses judgment but preserves world knowledge.

Format-specific terminal branches cause format-induced judgment inconsistencies.

Abstract

LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs. a True/False label). Existing diagnoses of these format-induced inconsistencies stop at the input-output level. Using Position-aware Edge Attribution Patching (PEAP), we causally investigate the internal mechanism in Gemma-3, Qwen2.5, and Llama-3. We find that judgments across structured understanding and open-ended preference tasks share a sparse, generalized Latent Evaluator sub-graph in the mid-to-late multi-layer perceptrons (MLPs); zero-ablating it collapses judgment while preserving world knowledge in architecturally modular models. By structurally decoupling abstract judging from output formatting, we provide a mechanistic account of format-induced inconsistency on the open-weight models we study: a continuous judgment signal computed in the shared trunk is mapped through fragile, format-specific terminal branches, enabling format-independent preference to be isolated downstream of the requested output format. Our findings imply that benchmark-level reliability comparisons across formats are partially measuring formatter geometry rather than evaluation quality.