RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

TL;DR

RubricEval introduces a comprehensive benchmark for assessing the accuracy of rubric-based evaluations of instruction-following in large language models, highlighting current limitations and avenues for improving judge reliability.

Contribution

It presents the first rubric-level meta-evaluation benchmark with diverse data, revealing the challenges and potential improvements in LLM judging accuracy.

Findings

GPT-4o achieves 55.97% on Hard subset

Rubric-level evaluation outperforms checklist-level evaluation

Explicit reasoning enhances judgment accuracy

Abstract

Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation.