RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation

TL;DR

RULERS introduces a framework that transforms natural language rubrics into executable specifications, enabling more reliable, stable, and scalable LLM evaluation by enforcing structured criteria, evidence verification, and calibration without model retraining.

Contribution

It presents RULERS, a novel compiler-executor system that converts rubrics into executable forms, addressing stability and alignment issues in LLM-based evaluation.

Findings

RULERS outperforms baselines in human agreement metrics.

It maintains stability against rubric perturbations.

Smaller models can match larger judges using RULERS.

Abstract

The LLM-as-a-Judge paradigm promises scalable rubric-based evaluation, yet aligning frozen black-box models with human standards remains a challenge due to inherent generation stochasticity. We reframe judge alignment as a criteria transfer problem and isolate three recurrent failure modes: rubric instability caused by prompt sensitivity, unverifiable reasoning that lacks auditable evidence, and scale misalignment with human grading boundaries. To address these issues, we introduce RULERS (Rubric Unification, Locking, and Evidence-anchored Robust Scoring), a compiler-executor framework that transforms natural language rubrics into executable specifications. RULERS operates by compiling criteria into versioned immutable bundles, enforcing structured decoding with deterministic evidence verification, and applying lightweight Wasserstein-based post-hoc calibration, all without updating model parameters. Extensive experiments on essay and summarization benchmarks demonstrate that RULERS significantly outperforms representative baselines in human agreement, maintains strong stability against adversarial rubric perturbations, and enables smaller models to rival larger proprietary judges. Overall, our results suggest that reliable LLM judging requires executable rubrics, verifiable evidence, and calibrated scales rather than prompt phrasing alone. Code is available at https://github.com/LabRAI/Rulers.git.