SedarEval: Automated Evaluation using Self-Adaptive Rubrics

TL;DR

SedarEval introduces a self-adaptive rubric-based evaluation paradigm for LLM outputs, creating detailed, question-specific scoring rubrics and training a specialized evaluator model that surpasses existing methods in accuracy and consistency.

Contribution

The paper presents a novel self-adaptive rubric framework and a comprehensive benchmark, SedarEval, with a trained evaluator LM that outperforms existing evaluation paradigms.

Findings

Evaluator LM achieves higher concordance with human grading than GPT-4.

SedarEval covers diverse domains including math, coding, and reasoning.

Self-adaptive rubrics improve evaluation precision and stability.

Abstract

The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.