iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision

TL;DR

iRULER introduces an intelligible, rubric-based framework for LLM evaluation that provides specific, justified, and actionable feedback, improving writing review and rubric creation through iterative refinement.

Contribution

The paper presents iRULER, a novel rubric-based approach for LLM evaluation that enhances feedback clarity, specificity, and user control, with iterative refinement of rubrics.

Findings

iRULER improves LLM-judged review scores in experiments.

Participants find iRULER more helpful and aligned than baseline methods.

Qualitative analysis confirms iRULER's adherence to design guidelines for feedback.

Abstract

Large Language Models (LLMs) have become indispensable for evaluating writing. However, text feedback they provide is often unintelligible, generic, and not specific to user criteria. Inspired by structured rubrics in education and intelligible AI explanations, we propose iRULER following identified design guidelines to \textit{scaffold} the review process by \textit{specific} criteria, providing \textit{justification} for score selection, and offering \textit{actionable} revisions to target different quality levels. To \textit{qualify} user-defined criteria, we recursively used iRULER with a rubric-of-rubrics to iteratively \textit{refine} rubrics. In controlled experiments on writing revision and rubric creation, iRULER most improved validated LLM-judged review scores and was perceived as most helpful and aligned compared to read-only rubric and text-based LLM feedback. Qualitative findings further support how iRULER satisfies the design guidelines for user-defined feedback. This work contributes interactive rubric tools for intelligible LLM-based review and revision of writing, and user-defined rubric creation.