JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

TL;DR

This paper introduces JudgeSense, a benchmark for evaluating prompt sensitivity in LLM-based judges, revealing that model size does not guarantee decision stability across paraphrased prompts.

Contribution

The paper presents JudgeSense, a comprehensive benchmark with hand-validated prompt paraphrases to systematically assess judge stability across tasks and models.

Findings

Coherence is the most distinguishing task for judge behavior.

Factuality judgments are generally stable under standard conditions.

Model scale does not reliably predict decision consistency.

Abstract

Large language models are widely adopted as automated evaluation judges, yet the stability of their verdicts under semantically equivalent prompt rephrasings remains largely unexamined. We conduct a systematic empirical study of prompt-induced decision instability across multiple evaluation tasks and judge architectures. To facilitate this analysis, we release JudgeSense, a benchmark comprising hand-validated prompt-paraphrase pairs spanning factuality, coherence, relevance, and preference, drawn from established NLP benchmarks and accompanied by comprehensive decision logs. The benchmark enables the measurement of judge stability across equivalent prompts, allowing researchers to assess whether stability correlates with model scale or instruction-tuning, and to identify which tasks are most sensitive to prompt wording. Our evaluation reveals that coherence remains the primary task for distinguishing judge behavior, while factuality judgments demonstrate high stability under standard conditions. Pairwise evaluation tasks consistently exhibit position bias. Crucially, we find that model scale is not a reliable proxy for consistency; notably, as an interesting result in our analysis, the largest and newest models are not the most consistent.