Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance

TL;DR

This paper introduces Think-with-Rubrics, a new approach where rubrics are internally generated and used as guidance during the reasoning process of language models, improving their performance on instruction following tasks.

Contribution

It transforms rubrics from external evaluators into internal reasoning guides within language models, enhancing their ability to follow instructions effectively.

Findings

Think-with-Rubrics outperforms the Rubric-as-Reward baseline by an average of 3.87 points.

Supervision from golden rubrics improves rubric quality and response consistency.

Self-generated rubrics increase internal coherence and model performance.

Abstract

Rubrics have been extensively utilized for evaluating unverifiable, open-ended tasks, with recent research incorporating them into reward systems for reinforcement learning. However, existing frameworks typically treat rubrics only as external evaluator disjointed from the policy's primary reasoning trace. Such design confines rubrics to post-hoc measurement, leaving them unable to actively guide the model's generation process. In this work, we introduce Think-with-Rubrics, a novel paradigm for instruction following tasks. Think-with-Rubrics integrates rubric generation into the reasoning context, transforming the rubric from an independent artifact into an internal guidance of LLM's generation. During training, LLM sequentially generates a rubric followed by a response, while a trained rubric verifier provides joint supervision by evaluating the consistency between the answer and the self-generated / golden rubrics. Experiments across multiple benchmarks demonstrate that Think-with-Rubrics consistently outperforms the Rubric-as-Reward baseline supervised by golden rubrics by an average of 3.87 points. We have also discussed the mechanism by which Think-with-Rubrics enhances model performance. Experimental results demonstrate that supervision from golden rubrics and self-generated rubrics enhances the performance of Think-with-Rubrics by improving the quality of self-generated rubrics and increasing the internal consistency of responses respectively.