Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior

TL;DR

This study reveals that LLM-based evaluators are individually consistent but differ systematically from each other, functioning as unique, stable 'fingerprints' rather than interchangeable judges, challenging assumptions of shared evaluation standards.

Contribution

The paper demonstrates that LLM evaluators have stable, individual-specific evaluation patterns, introducing the concept of 'evaluative fingerprints' and highlighting the non-interchangeability of judges.

Findings

Inter-judge agreement is near-zero (Krippendorff's α = 0.042).

Judges' evaluation patterns are highly distinguishable with up to 99.6% accuracy.

Judges' disagreement is structured and consistent, not random noise.

Abstract

LLM-as-judge systems promise scalable, consistent evaluation. We find the opposite: judges are consistent, but not with each other; they are consistent with themselves. Across 3,240 evaluations (9 judges x 120 unique video x pack items x 3 independent runs), inter-judge agreement is near-zero (Krippendorff's {\alpha} = 0.042). On two dimensions, judges disagree more than random noise would predict ({\alpha} < 0). Yet this disagreement isn't chaos; it's structured. A classifier identifies which judge produced an evaluation with 77.1% accuracy from rubric scores alone, rising to 89.9% with disposition features. Within model families, the signal is even stronger: GPT-4.1 and GPT-5.2 are distinguishable with 99.6% accuracy. We call this the reliability paradox: judges cannot agree on what constitutes quality, yet their disagreement patterns are so stable they function as fingerprints. Each judge implements a distinct, stable theory of quality: an "evaluative disposition" that shapes how it interprets any rubric. We characterize these dispositions along multiple axes: harshness/leniency, dimension emphasis, within-judge stability (ICC), and evidence behavior (receipt validity, semantic linkage via NLI, and shotgun index). The implication is stark: LLM judges are not interchangeable instruments measuring a shared construct. They are distinct measurement devices, each encoding its own implicit theory of quality. Averaging their scores produces a synthetic verdict that corresponds to no judge's actual values.